Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/aneeshdurg/multiversionpython
PoC of multiversioning in python
https://github.com/aneeshdurg/multiversionpython
Last synced: about 1 month ago
JSON representation
PoC of multiversioning in python
- Host: GitHub
- URL: https://github.com/aneeshdurg/multiversionpython
- Owner: aneeshdurg
- Created: 2024-01-31T15:08:35.000Z (12 months ago)
- Default Branch: main
- Last Pushed: 2024-02-01T00:12:57.000Z (12 months ago)
- Last Synced: 2024-02-02T01:24:59.312Z (11 months ago)
- Language: Python
- Size: 6.84 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# multiversionpython
The goal of this project is to enable having conflicting dependencies in python
by allowing multiple versions of the same library to be concurrently installed.
This is achieved by isolating every dependancy into it's own venv, and hooking
every `import` call to redirect them to importing within the appropriate env.
For example, consider the following dependency graph:
```
my_app
|
+--------+--------+
| |
v v
dep_a dep_b
| |
v v
pandas==2.2.0 pandas==2.0.3
```
Normally, this is not resolvable, and either `dep_a` or `dep_b` needs to be
patched or upgraded. Sometimes, such patching/upgrading is non-trivial, or
otherwise not desirable. This project demonstrates that using tools that are
already included with python, we can extend python's module management to allow
such dependency graphs.
The source code for intercepting imports and redirecting them to appropriate
venvs is in `importshim/`. A demo that shows how the example above might work is
in `demo/`.
This project is currently a proof of concept. With some work, maybe it could be
integrated into existing python tools and made more stable.
A more detailed explanation of what is happening is below:
+ tooling generates venvs + additional runtime importing facilities
+ each venv contains one dependency and it's sub-deps in isolation
+ importer will allow application to import libraries from each dependent venv
by means of generated config file.
+ importer will allow libraries to only import from within their isolated
environment by inspecting stack frames to determine the origin of `import`
calls.
The directory structure for a project might look like:
```
.
├── SOURCE_FILES...
├── pyproject.toml
├── app <-- This is the environment we want to install/run in
│ ├── bin
│ │ └── ...
│ ├── include
│ │ └── ...
│ ├── lib
│ │ └── python3.10
│ │ └── site-packages
│ │ └── importshim
│ │ ├── import_map.json <-- generated file
│ │ └── __init__.py
│ └── pyvenv.cfg
└── py_modules <-- entire directory is generated by tooling
├── pandas
│ ├── ...
│ ├── lib
│ │ └── python3.10
│ │ └── site-packages
│ │ └── ...
│ └── pyvenv.cfg
└── numpy
├── ...
├── lib
│ └── python3.10
│ └── site-packages
│ └── ...
└── pyvenv.cfg
```
here `py_modules` contains the isolated environments for each "top-level"
dependancy identified by `pyproject.toml`, `app` is the venv for the project,
containing only the generated import machinery, and `SOURCE_FILES` is the actual
project source code.
At present, there is no sharing of common sub-dependencies across modules.