Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/contextlab/davos
Import packages in Python, even if they aren't installed!
https://github.com/contextlab/davos
environment-management google-colab import install ipython jupyter package-management pip python reproducibility
Last synced: 24 days ago
JSON representation
Import packages in Python, even if they aren't installed!
- Host: GitHub
- URL: https://github.com/contextlab/davos
- Owner: ContextLab
- License: mit
- Created: 2021-03-02T23:08:46.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-10-27T04:22:56.000Z (about 1 year ago)
- Last Synced: 2024-10-12T11:21:54.005Z (24 days ago)
- Topics: environment-management, google-colab, import, install, ipython, jupyter, package-management, pip, python, reproducibility
- Language: TeX
- Homepage:
- Size: 36.2 MB
- Stars: 34
- Watchers: 4
- Forks: 8
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
README
davos
> _Someone once told me that the night is dark and full of terrors. And tonight I am no knight. Tonight I am Davos the
smuggler again. Would that you were an onion._
# Introduction
The `davos` library provides Python with an additional keyword: **`smuggle`**.
[The `smuggle` statement](#the-smuggle-statement) works just like the built-in [`import` statement](https://docs.python.org/3/reference/import.html), with two major differences:
1. You can `smuggle` a package _without installing it first_
2. You can `smuggle` a _specific version_ of a package
Taken together, these two enhancements to `import` provide a powerful system for developing and sharing reproducible code that works across different users and environments.
## Table of contents
- [Table of contents](#table-of-contents)
- [Introduction (↑)](#introduction)
- [Why would I want an alternative to `import`?](#why-would-i-want-an-alternative-to-import)
- [Why not use virtual environments, containers, and/or virtual machines instead?](#why-not-use-virtual-environments-containers-andor-virtual-machines-instead)
- [Quick start](#okay-so-how-do-i-use-this-thing)
- [Installation](#installation)
- [Latest Stable PyPI Release](#latest-stable-pypi-release)
- [Latest GitHub Update](#latest-github-update)
- [Installing in Colaboratory](#installing-in-colaboratory)
- [Overview](#overview)
- [Smuggling Missing Packages](#smuggling-missing-packages)
- [Smuggling Specific Package Versions](#smuggling-specific-package-versions)
- [Use Cases](#use-cases)
- [Simplify sharing reproducible code & Python environments](#simplify-sharing-reproducible-code--python-environments)
- [Guarantee your code always uses the latest version, release, or revision](#guarantee-your-code-always-uses-the-latest-version-release-or-revision)
- [Compare behavior across package versions](#compare-behavior-across-package-versions)
- [User guide](#usage)
- [The `smuggle` Statement](#the-smuggle-statement)
- [Syntax](#smuggle-statement-syntax)
- [Rules](#smuggle-statement-rules)
- [The Onion Comment](#the-onion-comment)
- [Syntax](#onion-comment-syntax)
- [Rules](#onion-comment-rules)
- [The `davos` Config](#the-davos-config)
- [Reference](#config-reference)
- [Top-level Functions](#top-level-functions)
- [How It Works: The `davos` Parser](#how-it-works-the-davos-parser)
- [Additional Notes](#additional-notes)
## Why would I want an alternative to `import`?
In many cases, `smuggle` and `import` do the same thing—*if you're
running code in the same environment you developed it in*. But what if you want
to share a [Jupyter notebook](https://jupyter.org/) containing your code with
someone else? If the user (i.e., the "someone else" in this example) doesn't
have all of the packages your notebook imports, Python will raise an exception
and the code won't run. It's not a huge deal, of course, but it's inconvenient
(e.g., the user might need to `pip`-install the missing packages, restart their
kernel, re-run the code up to the point it crashed, etc.—possibly going
through this cycle multiple times until the thing finally runs).
A second (and more subtle) issue arises when the developer (i.e., the person
who *wrote* the code) used or assumed different versions of the imported
packages than what the user has installed in their environment. So maybe the
original author was developing and testing their code using `pandas` 1.3.5, but
the user hasn't upgraded their `pandas` installation since 0.25.0. Python will
happily "`import pandas`" in both cases, but any changes across those versions
might change what the developer's code actually does in the user's (different)
environment—or cause it to fail altogether.
The problem `davos` tries to solve is similar to the idea motivating virtual
environments, containers, and virtual machines: we want a way of replicating
the original developer's environment on the user's machine, to a sufficiently
good approximation that we can be "reasonably confident" that the code will
continue to behave as expected.
When you `smuggle` packages instead of importing them, it guarantees (for
whatever environment the code is running in) that the packages are importable,
even if they hadn't been installed previously. Under the hood, `davos` figures
out whether the package is available, and if not, it uses `pip` to download and
install anything that's missing (including missing dependencies). From that
point, after having automatically handled those sorts of dependency issues,
`smuggle` behaves just like `import`.
The second powerful feature of `davos` comes from another construct, called
"[_onion comments_](#the-onion-comment)." These are like standard Python
comments, but they appear on the same line(s) as `smuggle` statements, and they
are formatted in a particular way. Onion comments provide a way of precisely
controlling how, when, and where packages are installed, how (or if) the system
checks for existing installations, and so on. A key feature is the ability to
specify exactly which version(s) of each package are imported into the current
workspace. When used in this way, `davos` enables authors to guarantee that the
same versions of the packages they developed their code with will also be
imported into the user's workspace at the appropriate times.
## Why not use virtual environments, containers, and/or virtual machines instead?
Psst-- we'll let you in on a little secret: importing `davos` *automatically*
creates a virtual environment for your notebook. However, whereas setting up a
virtual environment is usually left to the user, `davos` handles the pesky
details for you, without you needing to think about them. Any packages you
`smuggle` via `davos` that aren't available in the notebook's original runtime
environment are installed into a new virtual environment. This ensures that
`davos` will not change the runtime environment (e.g., by installing new
packages, changing existing package versions, etc.).
By default, each notebook's virtual environment is stored in a hidden ".davos"
folder inside the current user's home directory. The default environment name
is computed to uniquely identify each notebook, according to its filename and
path. However, a notebook's virtual environment may be customized by setting
`davos.project` to any string that can be used as a valid folder name in the
user's operating system. This is useful for multi-notebook projects that share
dependencies (without needing to duplicate each package installation for each
notebook).
If you prefer, you can also disable `davos`'s virtual environment
infrastructure by setting `davos.project` to `None`. Doing so will cause any
packages installed by `davos` to affect the notebook's runtime environment.
This is generally not recommended, as it can lead to unintended consequences
for other code that shares the runtime environment. That said, `davos` also
works great when used inside of (standard) virtual environments, containers,
and virtual machines.
There are a few additional specific advantages to `davos` that go beyond more
typical virtual environments, containers, and/or virtual machines. The main
advantage is that `davos` is very lightweight: importing `davos` into a
notebook-based environment unlocks all of its functionality without needed to
install, set up, and learn how to use additional stuff. There is none of the
typical overhead of setting up a new virtual environment (or container, virtual
machine, etc.), installing third-party tools, writing and sharing configuration
files, and so on. All of your code *and its dependencies* may be contained in a
single notebook file.
## Okay... so how do I use this thing?
To turn a standard [Jupyter (IPython) notebook](https://jupyter.org/), including a [Google Colaboratory notebook](https://colab.research.google.com), into a `davos`-enhanced notebook, just add two lines to the first cell:
```python
%pip install davos
import davos
```
This will enable the `smuggle` keyword in your notebook environment. Then you can do things like:
```python
# pip-install numpy v1.23.1, if needed
smuggle numpy as np # pip: numpy==1.23.1
# the smuggled package is fully imported and usable
arr = np.arange(15).reshape(3, 5)
# and the onion comment guarantees the desired version!
assert np.__version__ == '1.23.1'
```
Interested? Curious? Intrigued? Check out the table of contents for more
details! You may also want to check out our [paper](paper/main.pdf) for more
formal descriptions and explanations.
## Installation
### Latest Stable PyPI Release
[](https://pypi.org/project/davos/)
[]((https://pypi.org/project/davos/))
[]((https://pypi.org/project/davos/))
```sh
pip install davos
```
### Latest GitHub Update
[](https://github.com/ContextLab/davos/releases)
[](https://github.com/ContextLab/davos/commits/main)
[](https://github.com/ContextLab/davos/releases/latest)
```sh
pip install git+https://github.com/ContextLab/davos.git
```
### Installing in Colaboratory
To install `davos` in [Google Colab](https://colab.research.google.com/), add a new cell to the top of your notebook with an
percentage sign (`%`) followed by one of the commands above (e.g., `%pip install davos`). You'll likely also want to `import davos`,
which enables the `smuggle` syntax. Run the cell to install `davos` on the runtime virtual machine.
**Note**: restarting the Colab runtime does not affect installed packages. However, if the runtime is "factory reset"
or disconnected due to reaching its idle timeout limit, you'll need to rerun the cell to reinstall `davos` on the fresh
VM instance.
## Overview
The primary way to use `davos` is via [the `smuggle` statement](#the-smuggle-statement), which is made available
simply by running `import davos`. Like
[the built-in `import` statement](https://docs.python.org/3/reference/import.html), the `smuggle` statement is used to
load packages, modules, and other objects into the current namespace. The main difference between the two is in how
they handle missing packages and specific package versions.
### Smuggling Missing Packages
`import` requires that packages be installed _before_ the start of the interpreter session. Trying to `import` a package
that can't be found locally will throw a
[`ModuleNotFoundError`](https://docs.python.org/3/library/exceptions.html#ModuleNotFoundError), and you'll have to
install the package from the command line, restart the Python interpreter to make the new package importable, and rerun
your code in full in order to use it.
**The `smuggle` statement, however, can handle missing packages on the fly**. If you `smuggle` a package that isn't
installed locally, `davos` will install it for you, make its contents available to Python's
[import machinery](https://docs.python.org/3/reference/import.html), and load it into the namespace for immediate use.
You can control _how_ `davos` installs missing packages by adding a special type of inline comment called an
["onion" comment](#the-onion-comment) next to a `smuggle` statement.
### Smuggling Specific Package Versions
One simple but powerful use for [onion comments](#the-onion-comment) is making `smuggle` statements _version-sensitive_.
Python doesn't provide a native, viable way to ensure a third-party package imported at runtime matches a specific
version or satisfies a particular [version constraint](https://www.python.org/dev/peps/pep-0440/#version-specifiers).
Many packages expose their version info via a top-level `__version__` attribute (see
[PEP 396](https://www.python.org/dev/peps/pep-0396/)), and certain tools (such as the standard library's
[`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html) and
[`setuptools`](https://setuptools.readthedocs.io/en/latest/index.html)'s
[`pkg_resources`](https://setuptools.readthedocs.io/en/latest/pkg_resources.html)) attempt to parse version info from
installed distributions. However, using these to constrain imported package would require writing extra code to compare
version strings and _still_ manually installing the desired version and restarting the interpreter any time an
invalid version is caught.
Additionally, for packages installed through a version control system (e.g., [git](https://git-scm.com/)), this would be
insensitive to differences between revisions (e.g., commits) within the same semantic version.
**`davos` solves these issues** by allowing you to specify a specific version or set of acceptable versions for each
smuggled package. To do this, simply provide a
[version specifier](https://www.python.org/dev/peps/pep-0440/#version-specifiers) in an
[onion comment](#the-onion-comment) next to the `smuggle` statement:
```python
smuggle numpy as np # pip: numpy==1.23.1
from pandas smuggle DataFrame # pip: pandas>=1.0,<2.0
```
In this example, the first line will load [`numpy`](https://numpy.org/) into the local namespace under the alias "`np`",
just as "`import numpy as np`" would. First, `davos` will check whether `numpy` is installed locally, and if so, whether
the installed version _exactly_ matches `1.23.1`. If `numpy` is not installed, or the installed version is anything
other than `1.23.1`, `davos` will use the specified _installer program_, [`pip`](https://pip.pypa.io/en/stable/), to
install `numpy==1.23.1` before loading the package.
Similarly, the second line will load the "`DataFrame`" object from the [`pandas`](https://pandas.pydata.org/) library,
analogously to "`from pandas import DataFrame`". A local `pandas` version of `1.2.1` would be used, but a local version
of `2.1.1` would cause `davos` to replace it with a valid `pandas` version, as if you had manually run `pip install
pandas>=1.0,<2.0`.
In both cases, the imported versions will fit the constraints specified in their [onion comments](#the-onion-comment),
and the next time `numpy` or `pandas` is smuggled with the same constraints, valid local installations will be found.
You can also force the state of a smuggled packages to match a specific VCS ref (branch, revision, tag, release, etc.).
For example:
```python
smuggle hypertools as hyp # pip: git+https://github.com/ContextLab/hypertools.git@98a3d80
```
will load [`hypertools`](https://hypertools.readthedocs.io/en/latest/) (aliased as "`hyp`"), as the package existed
[on GitHub](https://github.com/ContextLab/hypertools), at commit
[98a3d80](https://github.com/ContextLab/hypertools/tree/98a3d80). The general format for VCS references in
[onion comments](#the-onion-comment) follows that of the
[`pip-install`](https://pip.pypa.io/en/stable/topics/vcs-support) command. See the
[notes on smuggling from VCS](#notes-vcs-smuggle) below for additional info.
And with [a few exceptions](#notes-c-extensions), smuggling a specific package version will work _even if the package
has already been imported_!
**Note**: `davos` v0.2.x supports [IPython](https://ipython.readthedocs.io/en/stable/) environments (e.g.,
[Jupyter](https://jupyter.org/) and [Colaboratory](https://colab.research.google.com/) notebooks) only. v0.3.x will add
support for "regular" (i.e., non-interactive) Python scripts.
### Use Cases
#### Simplify sharing reproducible code & Python environments
Different versions of the same package can often behave quite differently—bugs are introduced and fixed, features
are implemented and removed, support for Python versions is added and dropped, etc. Because of this, Python code that is
meant to be _reproducible_ (e.g., tutorials, demos, data analyses) is commonly shared alongside a set of fixed versions
for each package used. And since there is no Python-native way to specify package versions at runtime (see
[above](#smuggling-specific-package-versions)), this typically takes the form of a pre-configured development
environment the end user must build themselves (e.g., a [Docker](https://www.docker.com/) container or
[conda](https://docs.conda.io/en/latest/) environment), which can be cumbersome, slow to set up, resource-intensive, and
confusing for newer users, as well as require shipping both additional specification files _and_ setup instructions
along with your code. And even then, a well-intentioned user may alter the environment in a way that affects your
carefully curated set of pinned packages (such as installing additional packages that trigger dependency updates).
Instead, `davos` allows you to share code with one simple instruction: _just `pip install davos`!_ Replace your `import`
statements with `smuggle` statements, pin package versions in onion comments, and let `davos` take care of the rest.
Beyond its simplicity, this approach ensures your predetermined package versions are in place every time your code is
run.
#### Guarantee your code always uses the latest version, release, or revision
If you want to make sure you're always using the most recent release of a certain package, `davos` makes doing so easy:
```python
smuggle mypkg # pip: mypkg --upgrade
```
Or if you have an automation designed to test your most recent commit on GitHub:
```python
smuggle mypkg # pip: git+https://username/reponame.git
```
#### Compare behavior across package versions
The ability to `smuggle` a specific package version even after a different version has been imported makes `davos` a
useful tool for comparing behavior across multiple versions of the same package, within the same interpreter session:
```python
def test_my_func_unchanged():
"""Regression test for `mypkg.my_func()`"""
data = list(range(10))
smuggle mypkg # pip: mypkg==0.1
result1 = mypkg.my_func(data)
smuggle mypkg # pip: mypkg==0.2
result2 = mypkg.my_func(data)
smuggle mypkg # pip: git+https://github.com/MyOrg/mypkg.git
result3 = mypkg.my_func(data)
assert result1 == result2 == result3
```
## Usage
### The `smuggle` Statement
#### Syntax
The `smuggle` statement is meant to be used in place of
[the built-in `import` statement](https://docs.python.org/3/reference/import.html) and shares
[its full syntactic definition](https://docs.python.org/3/reference/simple_stmts.html#the-import-statement):
```ebnf
smuggle_stmt ::= "smuggle" module ["as" identifier] ("," module ["as" identifier])*
| "from" relative_module "smuggle" identifier ["as" identifier]
("," identifier ["as" identifier])*
| "from" relative_module "smuggle" "(" identifier ["as" identifier]
("," identifier ["as" identifier])* [","] ")"
| "from" module "smuggle" "*"
module ::= (identifier ".")* identifier
relative_module ::= "."* module | "."+
```
NB: uses the modified BNF grammar notation described in
The Python Language Reference,
here; see
here for the lexical definition
of identifier
In simpler terms, **any valid syntax for `import` is also valid for `smuggle`**.
#### Rules
- Like `import` statements, `smuggle` statements are whitespace-insensitive, unless a lack of whitespace between two
tokens would cause them to be interpreted as a different token:
```python
from os.path smuggle dirname, join as opj # valid
from os . path smuggle dirname ,join as opj # also valid
from os.path smuggle dirname, join asopj # invalid ("asopj" != "as opj")
```
- Any context that would cause an `import` statement _not_ to be executed will have the same effect on a `smuggle`
statement:
```python
# smuggle matplotlib.pyplot as plt # not executed
print('smuggle matplotlib.pyplot as plt') # not executed
foo = """
smuggle matplotlib.pyplot as plt""" # not executed
```
- Because the `davos` parser is less complex than the full Python parser, there are two fairly non-disruptive edge
cases where an `import` statement would be syntactically valid but a `smuggle` statement would not:
1. The [exec](https://docs.python.org/3.8/library/functions.html#exec) function
```python
exec('from pathlib import Path') # executed
exec('from pathlib smuggle Path') # raises SyntaxError
```
2. A one-line [compound statement](https://docs.python.org/3.9/reference/compound_stmts.html#compound-statements)
clause:
```python
if True: import random # executed
if True: smuggle random # raises SyntaxError
while True: import math; break # executed
while True: smuggle math; break # raises SyntaxError
for _ in range(1): import json # executed
for _ in range(1): smuggle json # raises SyntaxError
# etc...
```
- In [IPython](https://ipython.readthedocs.io/en/stable/) environments (e.g., [Jupyter](https://jupyter.org/) &
[Colaboratory](https://colab.research.google.com/) notebooks) `smuggle` statements always load names into the global
namespace:
```python
# example.ipynb
import davos
def import_example():
import datetime
def smuggle_example():
smuggle datetime
import_example()
type(datetime) # raises NameError
smuggle_example()
type(datetime) # returns
```
### The Onion Comment
An _onion comment_ is a special type of inline comment placed on a line containing a `smuggle` statement. Onion comments
can be used to control how `davos`:
1. determines whether the smuggled package should be installed
2. installs the smuggled package, if necessary
Onion comments are also useful when smuggling a package whose _distribution name_ (i.e., the name
used when installing it) is different from its _top-level module name_ (i.e., the name used when importing it). Take for
example:
```python
from sklearn.decomposition smuggle pca # pip: scikit-learn
```
The onion comment here (`# pip: scikit-learn`) tells `davos` that if "`sklearn`" does not exist
locally, the "`scikit-learn`" package should be installed.
#### Syntax
Onion comments follow a simple but specific syntax, inspired in part by the
[type comment syntax](https://www.python.org/dev/peps/pep-0484/#type-comments) introduced in
[PEP 484](https://www.python.org/dev/peps/pep-0484). The following is a loose (pseudo-)syntactic definition for an onion
comment:
```ebnf
onion_comment ::= "#" installer ":" install_opt* pkg_spec install_opt*
installer ::= ("pip" | "conda")
pkg_spec ::= identifier [version_spec]
```
NB: uses the modified BNF grammar notation described in
The Python Language Reference,
here; see
here for the lexical definition
of identifier
where `installer` is the program used to install the package; `install_opt` is any option accepted by the installer's
"`install`" command; and `version_spec` may be a
[version specifier](https://www.python.org/dev/peps/pep-0440/#version-specifiers) defined by
[PEP 440](https://www.python.org/dev/peps/pep-0440) followed by a
[version string](https://www.python.org/dev/peps/pep-0440/#public-version-identifiers), or an alternative syntax valid
for the given `installer` program. For example, [`pip`](https://pip.pypa.io/en/stable/) uses specific syntaxes for
[local](https://pip.pypa.io/en/stable/cli/pip_install/#local-project-installs),
[editable](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs), and
[VCS-based](https://pip.pypa.io/en/stable/topics/vcs-support) installation.
Less formally, **an onion comment simply consists of two parts, separated by a colon**:
1. the name of the installer program (e.g., [`pip`](https://pip.pypa.io/en/stable/))
2. arguments passed to the program's "install" command
Thus, you can essentially think of writing an onion comment as taking the full shell command you would run to install
the package, and replacing "_install_" with "_:_". For instance, the command:
```sh
pip install -I --no-cache-dir numpy==1.23.1 -vvv --timeout 30
```
is easily translated into an onion comment as:
```python
smuggle numpy # pip: -I --no-cache-dir numpy==1.23.1 -vvv --timeout 30
```
In practice, onion comments are identified as matches for the
[regular expression](https://en.wikipedia.org/wiki/Regular_expression):
```regex
#+ *(?:pip|conda) *: *[^#\n ].+?(?= +#| *\n| *$)
```
NB: support for installing smuggled packages via
conda will be added in v0.2. For v0.1,
"pip" should be used exclusively.
**Note**: support for installing smuggled packages via the [`conda`](https://docs.conda.io/en/latest/) package manager
will be added in v0.2. For v0.1, onion comments should always specify "`pip`" as the `installer` program.
#### Rules
- An onion comment must be placed on the same line as a `smuggle` statement; otherwise, it is not parsed:
```python
# assuming the dateutil package is not installed...
# pip: python-dateutil # <-- has no effect
smuggle dateutil # raises InstallerError (no "dateutil" package exists)
smuggle dateutil # raises InstallerError (no "dateutil" package exists)
# pip: python-dateutil # <-- has no effect
smuggle dateutil # pip: python-dateutil # installs "python-dateutil" package, if necessary
```
- An onion comment may be followed by unrelated inline comments as long as they are separated by at least one space:
```python
smuggle tqdm # pip: tqdm>=4.46,<4.60 # this comment is ignored
smuggle tqdm # pip: tqdm>=4.46,<4.60 # so is this one
smuggle tqdm # pip: tqdm>=4.46,<4.60# but this comment raises OnionArgumentError
```
- An onion comment must be the first inline comment immediately following a `smuggle` statement; otherwise, it is not
parsed:
```python
smuggle numpy # pip: numpy!=1.19.1 # <-- guarantees smuggled version is *not* v1.19.1
smuggle numpy # has no effect --> # pip: numpy==1.19.1
```
This also allows you to easily "comment out" onion comments:
```python
smuggle numpy ## pip: numpy!=1.19.1 # <-- has no effect
```
- Onion comments are generally whitespace-insensitive, but installer arguments must be separated by at least one space:
```python
from umap smuggle UMAP # pip: umap-learn --user -v --no-clean # valid
from umap smuggle UMAP#pip:umap-learn --user -v --no-clean # also valid
from umap smuggle UMAP # pip: umap-learn --user-v--no-clean # raises OnionArgumentError
```
- Onion comments have no effect on standard library modules:
```python
smuggle threading # pip: threading==9999 # <-- has no effect
```
- When smuggling multiple packages with a _single_ `smuggle` statement, an onion comment may be used to refer to the
**first** package listed:
```python
smuggle nilearn, nibabel, nltools # pip: nilearn==0.7.1
```
- If multiple _separate_ `smuggle` statements are placed on a single line, an onion comment may be used to refer to the
**last** statement:
```python
smuggle gensim; smuggle spacy; smuggle nltk # pip: nltk~=3.5 --pre
```
- For multiline `smuggle` statements, an onion comment may be placed on the first line:
```python
from scipy.interpolate smuggle ( # pip: scipy==1.6.3
interp1d,
interpn as interp_ndgrid,
LinearNDInterpolator,
NearestNDInterpolator,
)
```
... or on the last line:
```python
from scipy.interpolate smuggle (interp1d, # this comment has no effect
interpn as interp_ndgrid,
LinearNDInterpolator,
NearestNDInterpolator) # pip: scipy==1.6.3
```
... though the first line takes priority:
```python
from scipy.interpolate smuggle ( # pip: scipy==1.6.3 # <-- this version is installed
interp1d,
interpn as interp_ndgrid,
LinearNDInterpolator,
NearestNDInterpolator,
) # pip: scipy==1.6.2 # <-- this comment is ignored
```
... and all comments _not_ on the first or last line are ignored:
```python
from scipy.interpolate smuggle (
interp1d, # pip: scipy==1.6.3 # <-- ignored
interpn as interp_ndgrid,
LinearNDInterpolator, # unrelated comment # <-- ignored
NearestNDInterpolator
) # pip: scipy==1.6.2 # <-- parsed
```
- The onion comment is intended to describe how a specific smuggled package should be installed if it is not found
locally, in order to make it available for immediate use. Therefore, installer options that either (A) install
packages other than the smuggled package and its dependencies (e.g., from a specification file), or (B) cause the
smuggled package not to be installed, are disallowed. The options listed below will raise an `OnionArgumentError`:
- `-h`, `--help`
- `-r`, `--requirement`
- `-V`, `--version`
### The `davos` Config
The `davos` config object stores options and data that affect how `davos` behaves. After importing `davos`, the config
instance (a singleton) for the current session is available as `davos.config`, and its various fields are accessible as
attributes. The config object exposes a mixture of writable and read-only fields. Most `davos.config` attributes can be
assigned values to control aspects of `davos` behavior, while others are available for inspection but are set and used
internally. Additionally, certain config fields may be writable in some situations but not others (e.g. only if the
importing environment supports a particular feature). Once set, `davos` config options last for the lifetime of the
interpreter (unless updated); however, they do *not* persist across interpreter sessions. A full list of `davos` config
fields is available [below](#config-reference):
#### Reference
| Field | Description | Type | Default | Writable? |
| :---: | --- | :---: | :---: | :---: |
| `active` | Whether or not the `davos` parser should be run on subsequent input (cells, in Jupyter/Colab notebooks). Setting to `True` activates the `davos` parser, enables the `smuggle` keyword, and injects the `smuggle()` function into the user namespace. Setting to `False` deactivates the `davos` parser, disables the `smuggle` keyword, and removes "`smuggle`" from the user namespace (if it holds a reference to the `smuggle()` function). See [How it Works](#how-it-works) for more info. | `bool` | `True` | ✅ |
| `auto_rerun` | If `True`, when smuggling a previously-imported package that cannot be reloaded (see [Smuggling packages with C-extensions](#notes-c-extensions)), `davos` will automatically restart the interpreter and rerun all code up to (and including) the current `smuggle` statement. Otherwise, issues a warning and prompts the user with buttons to either restart/rerun or continue running. | `bool` | `False` | ✅ (**Jupyter notebooks only**) |
| `confirm_install` | Whether or not `davos` should require user confirmation (`[y/n]` input) before installing a smuggled package | `bool` | `False` | ✅ |
| `environment` | A label describing the environment into which `davos` was running. Checked internally to determine which interchangeable implementation functions are used, whether certain config fields are writable, and various other behaviors | `Literal['Python', 'IPython<7.0', 'IPython>=7.0', 'Colaboratory']` | N/A | ❌ |
| `ipython_shell` | The global IPython interactive shell instance | [`IPython.core`
`.interactiveshell`
`.InteractiveShell`](https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.interactiveshell.html#IPython.core.interactiveshell.InteractiveShell) | N/A | ❌ |
| `noninteractive` | Set to `True` to run `davos` in non-interactive mode (all user input and confirmation will be disabled). **NB**:
1. Setting to `True` disables `confirm_install` if previously enabled
2. If `auto_rerun` is `False` in non-interactive mode, `davos` will throw an error if a smuggled package cannot be reloaded | `bool` | `False` | ✅ (**Jupyter notebooks only**) |
| `pip_executable` | The path to the `pip` executable used to install smuggled packages. Must be a path (`str` or [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path)) to a real file. Default is programmatically determined from Python environment; falls back to `sys.executable -m pip` if executable can't be found | `str` | `pip` exe path or `sys.executable -m pip` | ✅ |
| `smuggled` | A cache of packages smuggled during the current interpreter session. Formatted as a `dict` whose keys are package names and values are the (`.split()` and `';'.join()`ed) onion comments. Implemented this way so that any non-whitespace change to installer arguments re-installation | `dict[str, str]` | `{}` | ❌ |
| `suppress_stdout` | If `True`, suppress all unnecessary output issued by both `davos` and the installer program. Useful when smuggling packages that need to install many dependencies and therefore generate extensive output. If the installer program throws an error while output is suppressed, both stdout & stderr will be shown with the traceback | `bool` | `False` | ✅ |
#### Top-level Functions
`davos` also provides a few convenience for reading/setting config values:
- **`davos.activate()`**
Activate the `davos` parser, enable the `smuggle` keyword, and inject the `smuggle()` function into the namespace.
Equivalent to setting `davos.config.active = True`. See [How it Works](#how-it-works) for more info.
- **`davos.deactivate()`**
Deactivate the `davos` parser, disable the `smuggle` keyword, and remove the name `smuggle` from the namespace if (and
only if) it refers to the `smuggle()` function. If `smuggle` has been overwritten with a different value, the variable
will not be deleted. Equivalent to setting `davos.config.active = False`. See [How it Works](#how-it-works) for more
- info.
- **`davos.is_active()`**
Return the current value of `davos.config.active`.
- **`davos.configure(**kwargs)`**
Set multiple `davos.config` fields at once by passing values as keyword arguments, e.g.:
```python
import davos
davos.configure(active=False, noninteractive=True, pip_executable='/usr/bin/pip3')
```
is equivalent to:
```python
import davos
davos.active = False
davos.noninteractive = True
davos.pip_executable = '/usr/bin/pip3'
```
## How It Works: The `davos` Parser
Functionally, importing `davos` appears to enable a new Python keyword, "_`smuggle`_". However, `davos` doesn't actually
modify the rules or [reserved keywords](https://docs.python.org/3/reference/lexical_analysis.html#keywords) used by
Python's parser and lexical analyzer in order to do so—in fact, modifying the Python grammar is not possible at
runtime and would require rebuilding the interpreter. Instead, in [IPython](https://ipython.readthedocs.io/en/stable/)
enivonments like [Jupyter](https://jupyter.org/) and
[Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb) notebooks, `davos` implements the `smuggle`
keyword via a combination of namespace injections and its own (far simpler) custom parser.
The `smuggle` keyword can be enabled and disabled at will by "activating" and "deactivating" `davos` (see the
[`davos` Config Reference](config-reference) and [Top-level Functions](#top-level-functions), above). When `davos` is
imported, it is automatically activated by default. Activating `davos` triggers two things:
1. The _`smuggle()` function_ is injected into the `IPython` user namespace
2. The _`davos` parser_ is registered as a
[custom input transformer](https://ipython.readthedocs.io/en/stable/config/inputtransforms.html)
IPython preprocesses all executed code as plain text before it is sent to the Python parser in order to handle
special constructs like [`%magic`](https://ipython.readthedocs.io/en/stable/interactive/magics.html) and
[`!shell`](https://ipython.readthedocs.io/en/stable/interactive/reference.html#system-shell-access) commands. `davos`
hooks into this process to transform `smuggle` statements into syntactically valid Python code. The `davos`
parser uses [this regular expression](https://github.com/ContextLab/davos/blob/main/davos/core/regexps.py) to match each
line of code containing a `smuggle` statement (and, optionally, an onion comment), extracts information from its text,
and replaces it with an analogous call to the _`smuggle()` function_. Thus, even though the code visible to the user may
contain `smuggle` statements, e.g.:
```python
smuggle numpy as np # pip: numpy>1.16,<=1.24 -vv
```
the code that is actually executed by the Python interpreter will not:
```python
smuggle(name="numpy", as_="np", installer="pip", args_str="""numpy>1.16,<=1.24 -vv""", installer_kwargs={'editable': False, 'spec': 'numpy>1.16,<=1.24', 'verbosity': 2})
```
The `davos` parser can be deactivated at any time, and doing so triggers the opposite actions of activating it:
1. The name "`smuggle`" is deleted from the `IPython` user namespace, *unless it has been overwritten and no longer
refers to the `smuggle()` function*
2. The `davos` parser input transformer is deregistered.
**Note**: in Jupyter and Colaboratory notebooks, IPython parses and transforms all text in a cell before sending it
to the kernel for execution. This means that importing or activating `davos` will not make the `smuggle` statement
available until the _next_ cell, because all lines in the current cell were transformed before the `davos` parser was
registered. However, _deactivating_ `davos` disables the `smuggle` statement immediately—although the `davos`
parser will have already replaced all `smuggle` statements with `smuggle()` function calls, removing the function from
the namespace causes them to throw `NameError`.
## Additional Notes
- **Reimplementing installer programs' CLI parsers**
The `davos` parser extracts info from onion comments by passing them to a (slightly modified) reimplementation of
their specified installer program's CLI parser. This is somewhat redundant, since the arguments will eventually be
re-parsed by the _actual_ installer program if the package needs to be installed. However, it affords a number of
advantages, such as:
- detecting errors early during the parser phase, before spending any time running code above the line containing the
`smuggle` statement
- preventing shell injections in onion comments—e.g., `#pip: --upgrade numpy && rm -rf /` fails due to the
`OnionParser`, but would otherwise execute successfully.
- allowing certain installer arguments to temporarily influence `davos` behavior while smuggling the current package
(see [Installer options that affect `davos` behavior](#notes-installer-opts) below for specific info)
- **Installer options that affect `davos` behavior**
Passing certain options to the installer program via an [onion comment](#the-onion-comment) will also affect the
corresponding `smuggle` statement in a predictable way:
- [**`--force-reinstall`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-force-reinstall) |
[**`-I`, `--ignore-installed`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-I) |
[**`-U`, `--upgrade`**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-I)
The package will be installed, even if it exists locally
- [**`--no-input`**](https://pip.pypa.io/en/stable/cli/pip/#cmdoption-no-input)
Disables input prompts, analogous to temporarily setting `davos.config.noninteractive` to `True`. Overrides value
of `davos.config.confirm_install`.
- [**`--src `**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-src) |
[**`-t`, `--target `**](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-t)
Prepends `` to [`sys.path`](https://docs.python.org/3/library/sys.html#sys.path) if not already present so
the package can be imported.
- **Smuggling packages with C-extensions**
Some Python packages that rely heavily on custom data types implemented via
[C-extensions](https://docs.python.org/3.9/extending/extending.html) (e.g., `numpy`, `pandas`) dynamically generate
modules defining various C functions and data structures, and link them to the Python interpreter when they are first
imported. Depending on how these objects are initialized, they may not be subject to normal garbage collection, and
persist despite their reference count dropping to zero. This can lead to unexpected errors when reloading the Python
module that creates them, particularly if their dynamically generated source code has been changed (e.g., because the
reloaded package is a newer version).
This can occasionally affect `davos`'s ability to `smuggle` a new version of a package (or dependency) that was
previously imported. To handle this, `davos` first checks each package it installs against
[`sys.modules`](https://docs.python.org/3.9/library/sys.html#sys.modules). If a different version has already been
loaded by the interpreter, `davos` will attempt to replace it with the requested version. If this fails, `davos` will
restore the old package version _in memory_, while replacing it with the new package version _on disk_. This allows
subsequent code that uses the non-reloadable module to still execute in most cases, while dependency checks for other
packages run against the updated version. Then, depending on the value of `davos.config.auto_rerun`, `davos` will
either either automatically restart the interpreter to load the updated package, prompt you to do so, or raise an
exception.
- **_`from` ... `import` ..._ statements and reloading modules**
The Python docs for [`importlib.reload()`](https://docs.python.org/3/library/importlib.html#importlib.reload) include
the following caveat:
> If a module imports objects from another module using
> [`from`](https://docs.python.org/3/reference/simple_stmts.html#from) …
> [`import`](https://docs.python.org/3/reference/simple_stmts.html#import) …, calling
> [`reload()`](https://docs.python.org/3/library/importlib.html#importlib.reload) for the other module does
> not redefine the objects imported from it — one way around this is to re-execute the `from` statement, another is to
> use `import` and qualified names (_module.name_) instead.
The same applies to smuggling packages or modules from which objects have already been loaded. If object _`name`_ from
module _`module`_ was loaded using either _`from module import name`_ or _`from module smuggle name`_, subsequently
running _`smuggle module # pip --upgrade`_ will in fact install and load an upgraded version of _`module`_, but the
the _`name`_ object will still be that of the old version! To fix this, you can simply run _`from module smuggle
name`_ either instead in lieu of or after _`smuggle module`_.
- **Smuggling packages from version control systems**
The first time during an interpreter session that a given package is installed from a VCS URL, it is assumed not to be
present locally, and is therefore freshly installed. `pip` clones non-editable VCS repositories into a temporary
directory, runs `setup.py install`, and then immediately deletes them. Since no information is retained about the
state of the repository at installation, it is impossible to determine whether an existing package satisfies the state
(i.e., branch, tag, commit hash, etc.) requested for smuggled package.
[comment]: <> (- As with _all_ code, you should use caution when running Python code containing `smuggle` statements that was not written by you or someone you know. )