{"id":15637580,"url":"https://github.com/python-morgan/morgan","last_synced_at":"2026-06-11T06:03:59.535Z","repository":{"id":59985667,"uuid":"530275193","full_name":"ido50/morgan","owner":"ido50","description":"PyPI Mirror for Restricted/Offline Environments","archived":false,"fork":false,"pushed_at":"2025-03-12T17:36:02.000Z","size":46,"stargazers_count":109,"open_issues_count":5,"forks_count":10,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-13T00:47:20.062Z","etag":null,"topics":["pip","pypi-packages","pypi-server","python","python3","repositories"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ido50.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["ido50"]}},"created_at":"2022-08-29T15:15:55.000Z","updated_at":"2025-03-21T14:58:11.000Z","dependencies_parsed_at":"2025-03-11T18:36:29.492Z","dependency_job_id":null,"html_url":"https://github.com/ido50/morgan","commit_stats":{"total_commits":19,"total_committers":5,"mean_commits":3.8,"dds":"0.26315789473684215","last_synced_commit":"4e612ebac226e846890a156ce2d8f817e6999bab"},"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ido50%2Fmorgan","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ido50%2Fmorgan/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ido50%2Fmorgan/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ido50%2Fmorgan/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ido50","download_url":"https://codeload.github.com/ido50/morgan/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650434,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pip","pypi-packages","pypi-server","python","python3","repositories"],"created_at":"2024-10-03T11:12:11.622Z","updated_at":"2026-06-11T06:03:59.529Z","avatar_url":"https://github.com/ido50.png","language":"Python","funding_links":["https://github.com/sponsors/ido50"],"categories":[],"sub_categories":[],"readme":"# Morgan\n\n**PyPI Mirror for Restricted/Offline Environments**\n\n## TOC\n\n\u003c!-- vim-markdown-toc GFM --\u003e\n\n* [Overview](#overview)\n* [Features and Concepts](#features-and-concepts)\n* [Installation](#installation)\n* [Usage](#usage)\n    * [Sample Configuration File](#sample-configuration-file)\n    * [Setting Up Multiple Mirrors](#setting-up-multiple-mirrors)\n    * [Mirroring From Other Indexes](#mirroring-from-other-indexes)\n* [How Does Morgan Resolve Dependencies?](#how-does-morgan-resolve-dependencies)\n* [Limitations](#limitations)\n* [Why Not Use X?](#why-not-use-x)\n* [Troubleshooting](#troubleshooting)\n* [License](#license)\n\n\u003c!-- vim-markdown-toc --\u003e\n\n## Overview\n\nMorgan is a PyPI mirror for restricted/offline networks/environments, where\naccess to the Internet is not available. It allows creating small mirrors that\ncan be used by multiple \"client\" Python environments (i.e. CPython versions,\noperating systems, etc.). The Morgan server is a single-file script that only\nuses modules the standard Python library, making it easy to deploy in such\nenvironments.\n\nThe basic idea is to run the mirror utility where Internet access is available—which\ngenerates a directory tree (\"package index\") with all of the required packages,\ntheir dependencies, and the server script—then copy this tree to the restricted\nnetwork (going through whatever security policies are in place), run the server\ninside the restricted network, and set `pip` in the client environments to use\nthe mirror instead of pypi.org, which they can't access anyway.\n\n## Features and Concepts\n\n- Runs under **Python 3.7 and up** (both server and mirrorer).\n- Package index is a **simple directory tree** that can be easily archived, copied,\n  rsynced, etc.\n- Package index contains a **configuration file** that lists **markers** for\n  different client **environments** and a list of **package requirement strings** as per [PEP 508](https://peps.python.org/pep-0508/).\n- Mirrorer automatically and **recursively mirrors dependencies** of all direct\n  requirements.\n- Only mirrors **optional dependencies** if they were part of the requirement\n  strings (a.k.a \"extras\"), or are relevant to the environment markers. This is\n  true for direct requirements and for dependencies of dependencies.\n- For each requirement, downloads the **latest version** that satisfies the\n  requirement (e.g. \"requests\u003e=2.24.0,\u003c2.27.0\" will download 2.26.0, whereas\n  \"requests\" will download the latest available version).\n- Downloads both **source distributions** and **binary distributions**. Only binary\n  distributions that are relevant to either of the environment markers defined in\n  the configuration file are downloaded.\n- **Server** is a one-file script with **no dependencies** outside the standard library\n  available in Python 3.7 and above. The script is automatically extracted into\n  the package index.\n- Server implements [PEP 503](https://peps.python.org/pep-0503/) (Simple Repository API),\n  [PEP 658](https://peps.python.org/pep-0658/) (Serve Distribution Metadata in the Simple Repository API),\n  and [PEP 691](https://peps.python.org/pep-0691/) (JSON-based Simple API for Python Package Indexes).\n\n## Installation\n\nMorgan is meant to be used as a command line utility (although it can be used\nas a library if necessary), therefore it is recommended to install via a utility\nsuch as [pipx](https://github.com/pypa/pipx):\n\n```sh\npipx install morgan\n```\n\nYou can also install it directly through `pip`:\n\n```sh\npython3 -m pip install morgan\n```\n\n## Usage\n\n1. Create a directory where the package index will reside.\n2. Create a \"morgan.ini\" file in this directory, with at least one environment\n   definition and list of requirements (see [Sample Configuration File](#sample-configuration-file) below).\n   You can use `morgan generate_env \u003e\u003e morgan.ini` to generate a configuration\n   block for the local interpreter. You can also use `morgan generate_reqs \u003e\u003e morgan.ini`\n   to generate list of requirements from all packages installed in the current\n   environment, which is especially useful when using virtual environments.\n3. Run the mirrorer from inside the package index via `morgan mirror` (alternatively,\n   provide the path of the package index via the `--index-path` flag).\n4. Copy the package index to the target environment, if necessary.\n5. Run the server using `python3 server.py`. Use `--help` for a full list of\n   flags and options. You can also use `morgan server` instead.\n\n### Sample Configuration File\n\nEnvironment configuration blocks can be automatically generated via\n`morgan generate_env`. I recommend you read the \"Environment Markers\" section of\n[PEP 508](https://peps.python.org/pep-0508/#environment-markers) to see exactly how they are calculated.\n\n```ini\n[env.legacy]\npython_version = 3.9\npython_full_version = 3.9.9\nos_name = posix\nplatform_tag = linux-x86_64\nsys_platform = linux\nplatform_machine = i686\nplatform_python_implementation = CPython\nplatform_system = Linux\nimplementation_name = cpython\n\n[env.edge]\nos_name = posix\nplatform_tag = linux-x86_64\nsys_platform = linux\nplatform_machine = x86_64\nplatform_python_implementation = CPython\nplatform_system = Linux\npython_version = 3.12\npython_full_version = 3.12.7\nimplementation_name = cpython\n\n[env.windows]\nos_name = nt\nplatform_tag = win_amd64\nsys_platform = win32\nplatform_machine = AMD64\nplatform_python_implementation = CPython\nplatform_system = Windows\npython_version = 3.12\npython_full_version = 3.12.7\nimplementation_name = cpython\n\n[requirements]\nrequests = \u003e=2.24.0\nprotobuf = ==3.20.1\nredis = \u003e4.1.0,\u003c4.2.1\nxonsh = [full]~=0.11.0\n```\n\nIn this example we can see two different client environments: a \"legacy\" one with\na relatively old installation of CPython 3.7.7 on 32-bit Linux, and an \"edge\"\nenvironment with a recent installation of CPython 3.10.6 on 64-bit Linux.\n\nAll these different markers are needed because they can be used in package\nmetadata, and Morgan needs them to determine which files and dependencies to\ndownload.\n\nThis configuration file sets \"requests\", \"protobuf\", \"redis\" and \"xonsh\" as the\npackages to mirror, with certain version specifications for each. The xonsh\nrequirements also specifies an extra which results in the downloading of certain\noptional dependencies. When the mirrorer is executed on a directory that contains this configuration file, it will find the\nlatest version that satisfies the specifications of each package, download\nrelevant files for each of them, and then recursively download _their_\ndependencies. It will download source files suitable for any environment, and\nbinary distributions (wheels) that match the definitions of the \"legacy\" and\n\"edge\" environments.\n\nConfiguration markers for a specific environment can easily be generated by\nrunning the provided `generate_env` command in the target environment.\n\n### Setting Up Multiple Mirrors\n\nYou can set up multiple PyPI mirrors by using separate configuration files.\nTo do so, simply add the `-c` or `--config` flag with the path of the\nconfiguration file to all relevant morgan commands. For example:\n\n```sh\nmorgan mirror --index-path /opt/pypi-mirror-1 --config /usr/local/etc/pypi-mirror-1.ini\nmorgan serve --index-path /opt/pypi-mirror-2 --config /usr/local/etc/pypi-mirror-2.ini\n```\n\n### Mirroring From Other Indexes\n\nTo mirror from a package index different than PyPI, simply provide the\n`-I` or `--index-url` flag with the URL of the index.\n\n## How Does Morgan Resolve Dependencies?\n\nDependency resolution in Python is hard. The standard for specifying dependencies\nwas only created in 2015 with [PEP 508](https://peps.python.org/pep-0508/). For years (and to this day), many projects\nhave used setuptools for distribution, which had its own mechanisms for\ndependency specifications. Many Python packages define their dependencies in\nnon-standard files such as Setuptools' requires.txt file, and not in their actual\nMETADATA files, despite the latter supporting dependency specification. Today,\nsetuptools has moved to using the newer pyproject.toml format standardized by\n[PEP 621](https://peps.python.org/pep-0621/).\n\nMany tools that attempt to extract requirements from package metadata, such as\n[pkginfo](https://pythonhosted.org/pkginfo/), do not return dependencies listed outside standard METADATA files.\n[pip-tools](https://pip-tools.readthedocs.io/) is the only tool outside of pip\nitself that can resolve dependencies from multiple sources, but like pip itself,\nit is not meant to be used as a library, and it is also meant to be used on\nproject development sources, not project distributions.\n\nMorgan, therefore, implements its own dependency resolution (which heavily relies\non the [packaging](https://pypi.org/project/packaging/) library). It utilizes an\nincremental strategy for extracting metadata from package distributions (whether\nsource or binary distributions), parsing any file that may contain necessary\nmetadata. Currently this includes METADATA, PKG-INFO, requires.txt,\nsetup_requires.txt and pyproject.toml files. Morgan also takes into account\nbuild dependencies to ensure that packages that are necessary to compile source\ndistributions are also available in the mirror (so, if a source distribution\nrequires [hatch](https://hatch.pypa.io/) in order to build, Morgan will ensure hatch is mirrored as\nwell).\n\nIt should be noted that recursively resolving dependencies is difficult, and\nrequires care in order to prevent circular dependencies. Morgan's current\nstrategy is very simple and possibly too broad, and although I haven't yet found\nany issues, they may happen. Please open a ticket if you encounter any such\nissues.\n\n## Limitations\n\n- Morgan currently only targets CPython. Packages/binary files that are specific\n  to other Python implementations are not mirrored. This may change in the\n  future.\n\n- Morgan only targets Python 3 packages. Python 2 packages are not mirrored and\n  there's no currently any plan to support them.\n\n- The only binary distributions supported are wheels. Eggs are not supported and\n  probably never will be.\n\n- The Morgan server is currently read-only. Packages cannot be published to the\n  index through it. This may change in the future.\n\n- Morgan does not mirror packages with versions that do not comply with\n  [PEP 440](https://peps.python.org/pep-0440/#version-scheme).\n  These are generally older packages that are no longer accepted by PyPI anyway.\n  This is unlikely to change.\n\n- Morgan does not mirror yanked files (see [PEP 592](https://peps.python.org/pep-0592/)). This is how PyPI supports\n  removing files without breaking projects that are specifically pinned to\n  them. This was a conscious decision that will probably not change, but\n  may be made configurable.\n\n## Why Not Use X?\n\nMorgan was written because of insufficiencies of other mirroring solutions:\n\n- [bandersnatch](https://github.com/pypa/bandersnatch/) is geared more towards mirroring of the entire PyPI repository,\n  with support for different filters to reduce/limit the size of the mirror.\n  Unfortunately, it doesn't automatically download dependencies of direct\n  requirements, making the \"package\" filter virtually unusable, and the \"platform\"\n  filters are not fine-grained enough. It also has many dependencies outside the\n  standard library.\n- [localshop](https://github.com/jazzband/localshop) is a proxy that basically caches PyPI responses to `pip` requests,\n  so it's not useful for restricted networks. It also has many non-standard-library\n  dependencies, and can't download binary distributions for multiple environments.\n- [devpi](https://www.devpi.net/) also works as a caching proxy, and also attempts to automatically refresh\n  from PyPI at regular intervals. It also has many non-standard-library\n  dependencies.\n- [pypickup](https://github.com/UB-Quantic/pypickup) was started around the same time as Morgan. It is more fitting for\n  an offline mirror than the above three but takes a different approach than Morgan. It\n  doesn't include a server, instead relying on `pip`'s ability to install from\n  a directory tree, so this tree needs to be accessible to your servers. It also\n  favors CLI commands over a configuration file (e.g. an `add` command is used to\n  mirror a dependency). I'm not sure how it resolves dependencies, if at all.\n\n## Troubleshooting\n\n- **pip fails installing a package file that exists.**\n\n  If `pip` fails with a message similar to \"ERROR: Could not find a version\n  that satisfies the requirement FastAPI (from versions: 0.88.0)\", despite\n  package files existing with the correct version and platform, it may be\n  that you're providing `pip` with a non-normalized name. You can either\n  use the normalized name when installing (lowercase the package name and\n  replace underscores with dashes, e.g. `pip install fastapi` instead of\n  `pip install FastAPI`), or start the server with the `--no-metadata` flag,\n  which will disable serving metadata files and workaround the issue. This seems\n  to be a non-standard behavior in `pip`, as it is sending the normalized name\n  to the server, but for some reason expecting to get the non-normalized name\n  back.\n\n## License\n\nMorgan is distributed under the terms of the [Apache License 2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpython-morgan%2Fmorgan","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpython-morgan%2Fmorgan","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpython-morgan%2Fmorgan/lists"}