https://github.com/anoopkcn/zenv

Python Environment Manager for HPC and Development Systems
https://github.com/anoopkcn/zenv

hpc hpc-clusters module package-manager pip python slurm supercomputing uv zenv

Last synced: 5 days ago
JSON representation

Python Environment Manager for HPC and Development Systems

Host: GitHub
URL: https://github.com/anoopkcn/zenv
Owner: anoopkcn
License: mit
Created: 2025-04-25T22:21:58.000Z (17 days ago)
Default Branch: main
Last Pushed: 2025-05-07T21:07:06.000Z (5 days ago)
Last Synced: 2025-05-07T21:08:06.936Z (5 days ago)
Topics: hpc, hpc-clusters, module, package-manager, pip, python, slurm, supercomputing, uv, zenv
Language: Zig
Homepage:
Size: 532 KB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# zenv

zenv: Python Environment Manager for HPC and Development Systems

`zenv` is a command-line tool written in Zig that manages Python virtual environments, primarily designed for High-Performance Computing (HPC) environments and development systems.

## Features

### Core Functionality

The tool provides several key features:

1. **Environment Management**: Create, activate, and manage Python virtual environments with configurations stored in `zenv.json`
2. **Registry**: Track environments globally so they can be activated from any directory
3. **System Targeting**: Configure environments for specific machines or clusters
4. **Dependency Management**: Install Python packages with awareness of what's already provided by system modules
5. **Module Integration**: Load HPC modules required for environment setup

## Installation

```sh
curl -fsSL https://raw.githubusercontent.com/anoopkcn/zenv/HEAD/install.sh | sh
```

Run the same command to update already insalled `zenv` version

### Alternative methods of installation

Manual download of stable release

```bash
# Replace with last stable release version:
curl -LO "https://github.com/anoopkcn/zenv/releases/download//zenv-x86_64-linux-musl-small.tar.gz"

# Extract the 'zenv' executable and move it somewhere in your 'PATH'
tar -xvf zenv-x86_64-linux-musl-small.tar.gz
mv zenv ~/.local/bin/
```

Build from Source

```bash
# Clone the repository
git clone https://github.com/anoopkcn/zenv.git

# Build the project
cd zenv
zig build

# Optional: Move executable to ~/.local/bin (assumes ~/.local/bin is in PATH)
mv zig-out/bin/zenv ~/.local/bin/
# OR
# Optional: Add to your PATH
export PATH="$PATH:path/to/zig-out/bin"
```

Check [release](https://github.com/anoopkcn/zenv/releases) for specific versions.
Supported OS: Linux(`aarch64`, `x86_64`), MacOS(`aarch64`, `x86_64`). _Windows support is not planned_

## Usage

### Initialize and setup an environment

```bash
zenv init
# This creates a `zenv.json` configuration file TEMPLATE in your project directory
# You can then modify this json file according to your needs and run:
zenv setup
# This will also register the environment to global ZENV_DIR/registry.json
```

Example of a `zenv.json` file:

```json
{
"test": {
"target_machines": ["jrlogin*.jureca", "*.juwels", "jrc*"],
"fallback_python": null,
"dependency_file": "requirements.txt",
"description": "Basic environment for jureca and juwels",
"modules": ["Stages/2025", "StdEnv", "Python", "CUDA"],
"dependencies": ["numpy>=1.20.0", "pandas"]
}
}
```

Provided `dependencies` will be installed in addition to the dependencies found in the optional `dependency_file` which can be (`requirements.txt` or a `pyproject.toml` file)

### Listing environments

List all environments registered for the current machine:

```bash
zenv list # for listing envs configured for current computer
# OR
zenv list --all # for listing all available envs in the registry
```

Example output:

```
- test
id : c3c494547b40f070b4c080f95c707622d84fe749
target : jureca, juwels, *
project : /path/to/project/
venv : /path/to/project/zenv/test
desc : Test python environment

Found 1 environment(s) for the current machine ('jrlogin01.jureca').
```

### Activating environments

Activate an environment by name or ID

Example:

```bash
# Activate by name
source $(zenv activate test)

# Activate by full ID
source $(zenv activate c3c494547b40f070b4c080f95c707622d84fe749)

# Activate by partial ID (first 7+ characters)
source $(zenv activate c3c4945)
```

### Registering and Deregistering Environments

A metadata information about the environments are stored at `ZENV_DIR/registry.json`. by default `ZENV_DIR` is `$HOME/.zenv`.
But one can set `ZENV_DIR` environment variable as any directory with write permission.

**This is done automatically when you run `zenv setup `**

```bash
zenv register
```

Remove an environment from the registry:

```bash
zenv deregister # Remove by name or ID
```

## Python Management

The default priority of the Python is as follows:

1. Module-provided Python (if HPC modules are loaded)
2. Explicitly configured 'fallback_python' from zenv.json (if not null)
3. zenv-managed pinned Python
4. System python3
5. System python

If you would like to use zenv-managed default Python for the environment, run:

```bash
# Install a python version if not done already
zenv python install

# Pin a specic python version
zenv python use

# use the pinned version
zenv setup --python
```

## Configuration Reference

One can have multiple environment configurations in the same `zenv.json` file and it supports the following structure:

```json
{
"base_dir": "",
"": {
"target_machines": [""],
"fallback_python": "",
"description": "",
"modules": ["", ""],
"dependency_file": "",
"dependencies": [""],
"setup_commands": [""],
"custom_activate_vars": {
"ENV_VAR_NAME": "value"
},
"activate_commands": [""]
},
"": {
"target_machines": [""]
}
}
```

In the configuration `target_machines` is required key(If you want, you can disable the validation check using `--no-host`), all other entries are optional. Top-level `base_dir` can be an absolute path or relative one(relative to the `zenv.json` file), if not provided it will create a directory called `zenv` at the project root. One can use wildcards to target specific systems, to mantch any machine use `*` or `any` (`"target_machines": ["*"]`). The lookup location of the `dependency_file` is the same directory as `zenv.json`.

## Help

```bash
zenv help
```

Output:

```
Usage: zenv [environment_name|id] [options]

Manages environments based on zenv.json configuration.

Commands:
init Create a new zenv.json template file in the current directory.

setup Set up the specified environment based on zenv.json file.
Creates a virtual environment in //.
and can be defined in the zenv.json file.

activate Output the path to the activation script.
You can use the environment name or its ID (full or partial).
use: source $(zenv activate ) to activate

cd Output the project directory path.
You can use the environment name or its ID (full or partial).
To change to the project directory, use:
cd $(zenv cd )

list List environments registered for the current machine.

list --all List all registered environments.

deregister Remove an environment from the global registry.
It does not remove the environment itself.

rm Deregister the environment AND remove its virtual
environment directory from the filesystem.

python Python management commands:
install : Install a specified Python version.
use : pinn a python version.
list : List all installed Python versions.

version, -v, --version Print the zenv version.

help, --help Show this help message.

Options for setup:
--no-host Bypass hostname validation, this is equivalant to
setting "target_machines": ["*"] in the zenv.json

--upgrade Attempt to upgrade the Python interpreter in an existing virtual
environment. If the environment doesn't exist or is corrupted,
it will be created fresh.

--python Use only the pinned Python set with 'use' subcommand.
This ignores the default python priority list.

--dev Install the current directory as an editable package.
Equivalent to running 'pip install --editable .'
Requires a valid 'setup.py' or 'pyproject.toml' in the directory.

--force It tries to install all dependencies even if they are already
provided by loaded modules.

Configuration (zenv.json):
The 'zenv.json' file defines your environments. Environment names occupy top level
"base_dir": "path/to/venvs", is exceptional top level key-value which specifies the
base directory for for storing environments. The value can be a relative path,
relative to zenv.json OR an absolute path(if path starts with a /).

Registry (ZENV_DIR/registry.json):
The global registry allows you to manage environments from any directory.
Setting up an environment will register that environment OR
register it with 'zenv register '. Once registred one can activate
using 'source $(zenv activate )' from any directory.

Python Priority list
1. Module-provided Python (if HPC modules are loaded)
2. Explicitly configured 'fallback_python' from zenv.json (if not null)
3. zenv-managed pinned Python
4. System python3
5. System python
This prority list can be ignored with 'zenv setup --python' which will use,
pinned python to manage the environement
```

## Issues

If you encounter any bugs open an Issue. To use the debug logging feature, users can set the `ZENV_DEBUG` environment variable:

Example:

```bash
ZENV_DEBUG=1 zenv setup
```

## License

[MIT License](LICENSE)

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/anoopkcn/zenv

Awesome Lists containing this project

README