https://github.com/stagedml/pylightnix

A Nix-style immutable data management library in Python
https://github.com/stagedml/pylightnix

immutable-infrastructure machine-learning nix package-management python

Last synced: about 1 year ago
JSON representation

A Nix-style immutable data management library in Python

• Host: GitHub
• URL: https://github.com/stagedml/pylightnix
• Owner: stagedml
• License: apache-2.0
• Created: 2020-01-09T11:16:27.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2024-03-14T23:09:52.000Z (over 2 years ago)
• Last Synced: 2025-02-17T09:38:59.349Z (over 1 year ago)
• Topics: immutable-infrastructure, machine-learning, nix, package-management, python
• Language: Python
• Homepage:
• Size: 910 KB
• Stars: 14
• Watchers: 5
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
![coverage](https://codecov.io/gh/stagedml/pylightnix/branch/master/graph/badge.svg)

Pylightnix

==========

Pylightnix is a minimalistic Python library for managing filesystem-based

immutable data, inspired by [Purely Functional Software Deployment Model thesis

by Eelco Dolstra](https://edolstra.github.io/pubs/phd-thesis.pdf) and the

[Nix](https://nixos.org) package manager.

The library may be thought as of low-level API for creating caching wrappers for

a subset of Python functions. In particular, Pylightnix allows user to

- Prepare (or, in our terms - **instantiate**) the computation plan aimed at

  creating a tree of linked immutable stage objects, stored in the filesystem.

- Implement (in our terms - **realize**) the prepared computation plan, access

  the resulting artifacts. Pylightnix is able to handle **non-deterministic**

  results of the computation.

- Handle changes in the computation plan, re-use the existing artifacts whenever

  possible.

- Gain full control over the cached data by inspecting the storage

``` python

import numpy as np

from pylightnix import *

fsinit('_pylightnix')

@autostage

def stage_dataset(A=100, file=[selfref, 'dataset.npy']):

  np.save(file, np.arange(0, A))

@autostage(nouts=10)

def stage_train(rindex:int, dataset=stage_dataset, model=[selfref, 'model.npy']):

  ds = np.load(dataset.file.val)

  np.save(model, rindex)

rrefs=realizeMany(instantiate(stage_train))

assert len(rrefs)>=10

```

## Documentation

QuickStart

[\[PDF\]](https://raw.github.com/stagedml/pylightnix-docs/master/Pylightnix-QuickStart-latest.pdf)

\| API Referece [\[MD\]](./docs/Reference.md)

Demos:

- [GNU Hello](./docs/demos/HELLO.md), turn Pylightnix into a toy

  package manager to build the GNU Hello from sources.

- [MDRUN](./docs/demos/MDRUN.py), evaluate code sections of a Markdown

  document, cache the results between runs.