{"id":15359836,"url":"https://github.com/matthewfeickert/cvmfs-venv","last_synced_at":"2025-10-25T01:35:09.397Z","repository":{"id":43660999,"uuid":"457931865","full_name":"matthewfeickert/cvmfs-venv","owner":"matthewfeickert","description":"Example implementation of getting a Python virtual environment to work with CVMFS LCG views","archived":false,"fork":false,"pushed_at":"2024-06-14T06:54:55.000Z","size":83,"stargazers_count":10,"open_issues_count":4,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-12T23:52:26.548Z","etag":null,"topics":["cvmfs","hep","hep-ex","lcg","physics","python","python-venv","python3"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/matthewfeickert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-02-10T20:11:23.000Z","updated_at":"2025-02-28T18:24:11.000Z","dependencies_parsed_at":"2023-11-18T08:26:18.959Z","dependency_job_id":"f2450eb6-8c81-4bb2-b171-89ba2079580c","html_url":"https://github.com/matthewfeickert/cvmfs-venv","commit_stats":{"total_commits":46,"total_committers":1,"mean_commits":46.0,"dds":0.0,"last_synced_commit":"563ad4cbf82b21e4189c4485e96e2ffec3e396e0"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewfeickert%2Fcvmfs-venv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewfeickert%2Fcvmfs-venv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewfeickert%2Fcvmfs-venv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/matthewfeickert%2Fcvmfs-venv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/matthewfeickert","download_url":"https://codeload.github.com/matthewfeickert/cvmfs-venv/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249024718,"owners_count":21200172,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvmfs","hep","hep-ex","lcg","physics","python","python-venv","python3"],"created_at":"2024-10-01T12:46:31.509Z","updated_at":"2025-10-25T01:35:09.392Z","avatar_url":"https://github.com/matthewfeickert.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cvmfs-venv\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7751033.svg)](https://doi.org/10.5281/zenodo.7751033)\n\nSimple command line utility for getting a Python virtual environment to work with CVMFS [LCG views][LCG_info]. This is done by adding additional hooks to the Python virtual environment's `bin/activate` script.\n\n[LCG_info]: https://lcginfo.cern.ch/\n\n## Install\n\nEither clone the repo to your directory or simply download the relevant files and place on `PATH`\n\n```console\n$ mkdir -p ~/.local/bin\n$ export PATH=~/.local/bin:\"${PATH}\"  # If ~/.local/bin not on PATH already\n$ curl -sL https://raw.githubusercontent.com/matthewfeickert/cvmfs-venv/main/cvmfs-venv.sh -o ~/.local/bin/cvmfs-venv\n$ chmod +x ~/.local/bin/cvmfs-venv\n```\n\n## Use\n\nSource the script to create a Python 3 virtual environment that can coexist with a CVMFS LCG view. The default name is `venv`.\n\n```console\n$ cvmfs-venv --help\nUsage: cvmfs-venv [-s|--setup] [--no-system-site-packages] [--no-update] [--no-uv] \u003cvirtual environment name\u003e\n\nOptions:\n -h --help      Print this help message\n -s --setup     String of setup options to be parsed\n --no-system-site-packages\n                The venv module '--system-site-packages' option is used by\n                default. While it is not recommended, this behavior can be\n                disabled through use of this flag.\n --no-update    After venv creation don't update pip and setuptools to the\n                latest releases. Use of this option is not recommended,\n                but is faster.\n --no-uv        After venv creation don't install uv and use it to update pip,\n                and setuptools. By default, uv is installed.\n\nNote: cvmfs-venv extends the Python venv module and so requires Python 3.3+.\n\nExamples:\n\n    * Create a Python 3 virtual environment named 'lcg-example' with the Python\n    runtime provided by LCG view 105 on AlmaLinux 9.\n\n        setupATLAS -3\n        lsetup 'views LCG_105 x86_64-el9-gcc12-opt'\n        cvmfs-venv lcg-example\n        . lcg-example/bin/activate\n\n    * Create a Python 3 virtual environment named 'atlas-ab-example' with the\n    Python runtime provided by ATLAS AnalysisBase release v25.2.15.\n\n        setupATLAS -3\n        asetup AnalysisBase,25.2.15\n        cvmfs-venv atlas-ab-example\n        . atlas-ab-example/bin/activate\n\n    * Create a Python 3 virtual environment named 'venv' with whatever Python\n    runtime \"$(command -v python3)\" evaluates to.\n\n        cvmfs-venv\n        . venv/bin/activate\n\n    * Setup LCG view 105 on AlmaLinux 9 and create a Python virtual environment\n    named 'lcg-example' using the Python 3.9 runtime it provides.\n\n        . cvmfs-venv --setup \"lsetup 'views LCG_105 x86_64-el9-gcc12-opt'\" lcg-example\n\n    * Setup ATLAS AnalysisBase release v25.2.15 and create a Python virtual\n    environment named 'atlas-ab-example' using the Python 3.9 runtime it\n    provides.\n\n        . cvmfs-venv --setup 'asetup AnalysisBase,25.2.15' atlas-ab-example\n```\n\n### Example: Virtual environment with LCG view\n\n```console\n$ ssh lxplus\n[feickert@lxplus924 ~]$ mkdir -p ~/.local/bin\n[feickert@lxplus924 ~]$ export PATH=~/.local/bin:\"${PATH}\"\n[feickert@lxplus924 ~]$ curl -sL https://raw.githubusercontent.com/matthewfeickert/cvmfs-venv/main/cvmfs-venv.sh -o ~/.local/bin/cvmfs-venv\n[feickert@lxplus924 ~]$ chmod +x ~/.local/bin/cvmfs-venv\n[feickert@lxplus924 ~]$ setupATLAS -3 --quiet\n[feickert@lxplus924 ~]$ lsetup 'views LCG_105 x86_64-el9-gcc12-opt'\n************************************************************************\nRequested:  views ...\n Setting up views LCG_105:x86_64-el9-gcc12-opt ...\n\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e\u003e Information for user \u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\u003c\n************************************************************************\n[feickert@lxplus924 ~]$ cvmfs-venv lcg-example\n# Creating new Python virtual environment 'lcg-example'\n[feickert@lxplus924 ~]$ . lcg-example/bin/activate\n(lcg-example) [feickert@lxplus924 ~]$ python -m pip list --local  # Show installed defaults\nPackage    Version\n---------- -------\npip        24.0\nsetuptools 69.5.1\n(lcg-example) [feickert@lxplus924 ~]$ python -m pip show hepdata-lib  # Still have full LCG view\nName: hepdata-lib\nVersion: 0.12.0\nSummary: Library for getting your data into HEPData\nHome-page: https://github.com/HEPData/hepdata_lib\nAuthor: Andreas Albert, Clemens Lange\nAuthor-email: hepdata-lib@cern.ch\nLicense: UNKNOWN\nLocation: /cvmfs/sft.cern.ch/lcg/views/LCG_105/x86_64-el9-gcc12-opt/lib/python3.9/site-packages\nRequires: future, hepdata-validator, numpy, PyYAML\nRequired-by:\n(lcg-example) [feickert@lxplus924 ~]$ uv pip install --upgrade awkward\n(lcg-example) [feickert@lxplus924 ~]$ python\nPython 3.9.12 (main, Jul 11 2023, 14:44:04)\n[GCC 12.1.0] on linux\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n\u003e\u003e\u003e import ROOT\n\u003e\u003e\u003e import XRootD\n\u003e\u003e\u003e import awkward\n\u003e\u003e\u003e exit()\n(lcg-example) [feickert@lxplus924 ~]$ python -m pip show awkward  # Get version installed in venv\nName: awkward\nVersion: 2.6.4\nSummary: Manipulate JSON-like data with NumPy-like idioms.\nHome-page:\nAuthor:\nAuthor-email: Jim Pivarski \u003cpivarski@princeton.edu\u003e\nLicense: BSD-3-Clause\nLocation: /afs/cern.ch/user/f/feickert/lcg-example/lib/python3.9/site-packages\nRequires: awkward-cpp, fsspec, importlib-metadata, numpy, packaging, typing-extensions\nRequired-by: cabinetry, coffea, servicex, uproot_browser\n(lcg-example) [feickert@lxplus924 ~]$ python -m pip list --local  # View of virtual environment controlled packages\nPackage            Version\n------------------ --------\nawkward            2.6.4\nawkward-cpp        33\nfsspec             2024.3.1\nimportlib_metadata 7.1.0\nnumpy              1.26.4\npackaging          24.0\npip                24.0\nsetuptools         69.5.1\ntyping_extensions  4.11.0\nzipp               3.18.1\n(lcg-example) [feickert@lxplus924 ~]$ uv pip list  # uv will show the same view\nPackage            Version\n------------------ --------\nawkward            2.6.4\nawkward-cpp        33\nfsspec             2024.3.1\nimportlib-metadata 7.1.0\nnumpy              1.26.4\npackaging          24.0\npip                24.0\nsetuptools         69.5.1\ntyping-extensions  4.11.0\nzipp               3.18.1\n(lcg-example) [feickert@lxplus924 ~]$ deactivate  # Resets PYTHONPATH given added hooks\n[feickert@lxplus924 ~]$ python -m pip show awkward  # Get CVMFS's old version\nName: awkward\nVersion: 1.10.3\nSummary: Manipulate JSON-like data with NumPy-like idioms.\nHome-page: https://github.com/scikit-hep/awkward-1.0\nAuthor: Jim Pivarski\nAuthor-email: pivarski@princeton.edu\nLicense: BSD-3-Clause\nLocation: /cvmfs/sft.cern.ch/lcg/views/LCG_105/x86_64-el9-gcc12-opt/lib/python3.9/site-packages\nRequires: numpy, packaging\nRequired-by: cabinetry, coffea, servicex, uproot_browser\n```\n\n## Dependencies\n\n`cvmfs-venv` has no dependencies beyond the ones it aims to extend: A Linux operating system that has CVMFS installed on it with a Python 3.3+ runtime with a functioning [`venv` module][venv docs].\n\nA full listing of all programs used outside of Bash shell builtins are:\n* `cat`\n* `curl`\n* `ed` or `vi`\n* `find`\n* `readlink`\n* `sed`\n* Python 3.3+ with `pip`\n\n## Why is this needed?\n\nWhen an LCG view or an ATLAS computing environment that uses software from CVFMS is setup, it manipulates and alters the [`PYTHONPATH` environment variable][PYTHONPATH docs].\nBy placing the contents of all the installed software of an LCG view or ATLAS release onto `PYTHONPATH` for the rest of the shell session, the protections and isolation of a Python virtual environment are broken.\nIt is not possible to fix this in a reliable and robust way that will not break the access of other software in the LCG view or ATLAS environment dependent on the Python packages in them.\nThe best that can be done is to control the directory tree at the **head** of `PYTHONPATH` in a stable manner that allows for most of the benefits of a Python virtual environment (control of install and versions of packages, isolation of directory tree).\n\nWhile [`lcgenv`][lcgenv] allows for package specific environment building, it still lacks the control to specify arbitrary versions of Python packages and will load additional libraries beyond what is strictly required by the target package dependency requirements.\nThat being said, if you are able to use an LCG view or `lcgenv` without any additional setup, you may not have need of specifying a Python virtual environment.\n\nWhile Python's [`venv` module][venv docs] does have the `--system-site-packages` option to\n\n\u003e Give the virtual environment access to the system site-packages dir.\n\nthis unfortunately isn't _quite_ enough.\nIt does allow for isolation to work, but the manipulation of `PYTHONPATH` makes it so that while packages can be _installed_ properly in the local virtual environment and will show up with `python -m pip list` if there is another version of that package provided by the already setup environment that package version's location on `PYTHONPATH` will take precedence.\nUsing `--system-site-packages` without `cvmfs-venv` is arguably even worse as it provides confusing differences in information between what `pip` has purported to install in the user's virtual environment and the user `pip list` view and the runtime environment.\n\n**Caveat**: This is an LCG view specific issue mostly. If **nothing from LCG is used** (like a pure ATLAS AnalysisBase environment, or an environment in a Linux container) then `--system-site-packages` by itself should be sufficient.\n\n## How things work\n\n`cvmfs-venv` provides a shim layer to manage activation and use of a Python virtual environment created with LCG view resources.\nIt does this by copying the structure of the `activate` scripts generated by Python's [`venv` module][venv docs].\nWhen `venv` creates a virtual environment it generates a shell script under the virtual environment's directory tree at `bin/activate`.\nThis `activate` script controls and edits the shell environmental variables `PATH` and [`PYTHONHOME`][PYTHONHOME docs] \u0026mdash; placing the virtual environment's `bin/` directory onto `PATH` and unsetting `PYTHONHOME` upon activation, and restoring their original values when `deactivate` is run.\n`cvmfs-venv` simply extends this existing behavior to also place the virtual environment's `site-packages/` directory onto `PYTHONPATH` during activation and to remove it on deactivation.\nThis is done by injecting Bash snippets directly into the `bin/activate` script generated by `venv` at positions found relative to the manipulation of `PYTHONHOME`.\n\n###  Advantages\n\n* As `cvmfs-venv` is just altering the contents of the `venv` virtual environment's `bin/activate` script it is extending existing functionality and not trying to remake virtual environments.\n* Once the virtual environment is setup and modified there is no additional dependency on the `cvmfs-venv` script that generated it.\n   - While it saves time it is not needed. You can setup the environment again without it.\n   ```console\n   $ . cvmfs-venv --setup \"lsetup 'views LCG_105 x86_64-el9-gcc12-opt'\" venv\n   ```\n   vs.\n   ```console\n   $ setupATLAS -3\n   $ lsetup \"views LCG_105 x86_64-el9-gcc12-opt\"\n   $ . venv/bin/activate\n   ```\n* As the virtual environment  is **prepended** to `PYTHONPATH` all packages installed in the virtual environment are automatically given higher precedence over existing packages of the same name found in the LCG view.\n   - If a package named `awkward` is found in the `venv` virtual environment and in the LCG view, `import awkward` with import it from the virtual environment.\n* Python packages not installed in the `venv` virtual environment but installed in the LCG view (e.g. ROOT, XRootD) can still be accessed inside of the virtual environment.\n   - **N.B.:** This is considered a \"advantage\" loosely, as it is only happening as a side effect of the isolation of the virtual environment being broken by the LCG view's `PYTHONPATH` manipulation.\n* Through additional manipulation of `PYTHONPATH` and `PATH` with the added `cvmfs-venv-rebase` functionality, any software added to `PATH` or `PYTHONPATH` (e.g., by CVMFS or ATLAS software) while the virtual environment is active will persist on `PATH` or `PYTHONPATH` when the virtual environment is deactivated, allowing for a smoother interactive experience.\n   - Example: If you want to use [`rucio`][rucio-site] both inside and outside of the virtual environment you can set it up either before sourcing the `bin/activate` script\n\n   ```console\n   $ setupATLAS -3\n   $ lsetup \"views LCG_105 x86_64-el9-gcc12-opt\"\n   $ lsetup \"rucio -w\"  # PYTHONPATH is altered by lsetup\n   $ command -v rucio  # rucio is found\n   $ . venv/bin/activate\n   (venv) $ command -v rucio  # rucio is found\n   (venv) $ deactivate\n   $ command -v rucio  # rucio is found\n   ```\n\n   or after, as `cvmfs-venv-rebase` will update path variables and keep the virtual environment's directory trees at the heads.\n\n   ```console\n   $ setupATLAS -3\n   $ lsetup \"views LCG_105 x86_64-el9-gcc12-opt\"\n   $ . venv/bin/activate\n   (venv) $ lsetup \"rucio -w\"  # PYTHONPATH is altered by lsetup\n   (venv) $ command -v rucio  # rucio is found\n   (venv) $ deactivate  # deactivate calls cvmfs-venv-rebase to persist non-venv paths\n   $ command -v rucio  # rucio is found\n   ```\n\n### Disadvantages\n\n* The isolation of a Python virtual environment is not recovered.\nPython packages installed in the LCG view can still be accessed inside of the virtual environment.\nThis can result in packages from the LCG view meeting requirements of other dependencies during a package install or upgrade (depending on the [upgrade strategy][pip-docs-upgrade-strategy]) and installing older versions then expected.\n   - Suggestion: When installing or upgrading with `pip` use the [`--ignore-installed` flag][pip-docs-ignore-installed].\n   - Suggestion: When using [`pip list`][pip-docs-list] to inspect installed packages, use the `--local` flag to not list packages from the LCG view.\n   ```\n   (venv) $ python -m pip list --local\n   ```\n* As the Python version tied to the virtual environment is provided by LCG view, any changes to the LCG view version require setting up the Python virtual environment from scratch.\n   - It is highly advised that your environment be controlled with strict `requirements.txt` and lock files (e.g. generated from [`pip-tools`][pip-tools] using `pip-compile`) making it reproducible and easy to setup again as a consequence.\n   - Example:\n   ```console\n   (venv) $ python -m pip install --upgrade pip-tools\n   (venv) $ echo \"scipy==1.11.3\" \u003e requirements.txt  # define high level requirements\n   (venv) $ pip-compile --generate-hashes --output-file requirements.lock requirements.txt  # Generate full environment lock file\n   (venv) $ python -m pip install --no-deps --require-hashes --only-binary :all: --requirement requirements.lock  # secure-install for reproducibility\n   ```\n* Having all of the environment manipulation happen inside of the `venv`'s `bin/activate` script means that the virtual environment needs to be activated after any LCG view or ATLAS software (which make `PYTHONPATH` not empty) to trigger `PYTHONPATH` manipulation.\nThis essentially means that the virtual environment must not be activated first in any setup script.\n\n## Citation\n\nThe preferred BibTeX entry for citation of `cvmfs-venv` is\n\n```\n@software{cvmfs-venv,\n  author = {Matthew Feickert},\n  title = \"{cvmfs-venv: v0.0.7}\",\n  version = {0.0.7},\n  doi = {10.5281/zenodo.7751033},\n  url = {https://doi.org/10.5281/zenodo.7751033},\n  note = {https://github.com/matthewfeickert/cvmfs-venv/releases/tag/v0.0.7}\n}\n```\n\n[PYTHONHOME docs]: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHOME\n[PYTHONPATH docs]: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH\n[venv docs]: https://docs.python.org/3/tutorial/venv.html\n\n[pip-docs-list]: https://pip.pypa.io/en/stable/cli/pip_list/#pip-list\n[pip-docs-upgrade-strategy]: https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-upgrade-strategy\n[pip-docs-ignore-installed]: https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-I\n\n[pip-tools]: https://github.com/jazzband/pip-tools\n\n[lcgenv]: https://gitlab.cern.ch/GENSER/lcgenv\n[rucio-site]: https://rucio.cern.ch/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatthewfeickert%2Fcvmfs-venv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatthewfeickert%2Fcvmfs-venv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatthewfeickert%2Fcvmfs-venv/lists"}