Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/vahidzee/dypy

Toolkit for Dynamic Python code manipulations
https://github.com/vahidzee/dypy

dynamic-code-execution dynamic-objects functional-programming lazy-evaluation

Last synced: 9 days ago
JSON representation

Toolkit for Dynamic Python code manipulations

Awesome Lists containing this project

README 

        
# Dynamic Python (DyPy): a toolset for dynamic python code manipulations



Have you ever found yourself coding up boilerplates to handle different scenarios? It's most likely that you have thought about using python's `eval` or `exec` in order to decouple some part of your code to be specified and evaluated later on at run-time. But, you have probably also come accross people who discourage you from using `eval`/`exec`. This is because programs with `eval`/`exec` are considered vulnarable and can be used to execute malicious code.



While this is true in general, but in many cases, you do not care about security concerns, and your highest priority is implementing a quick solution to a general problem which could be solved by using `eval` or `exec`. This is where dypy comes in. It allows you to use `eval` and `exec` effectively by providing utilities to dynamically compile code, lookup variables and lazily evaluate them.



**Table of Contents**

- [Dynamic Python (DyPy): a toolset for dynamic python code manipulations](#dynamic-python-dypy-a-toolset-for-dynamic-python-code-manipulations)

  - [Installation](#installation)

  - [Usage](#usage)

    - [Dynamic Evaluation](#dynamic-evaluation)

      - [Variable Lookup](#variable-lookup)

      - [Dynamic Function Evaluation](#dynamic-function-evaluation)

    - [Variable Assignment](#variable-assignment)

  - [License](#license)

  - [Citation](#citation)



## Installation



```bash

pip install dypy

```



## Usage



### Dynamic Evaluation



You can use `dypy.eval` to combine the functionality of `dypy.eval_function` (see [here](#dynamic-function-evaluation)) and `dypy.get_value` (see [here](#variable-lookup)). You can do as follows:



```python

import dypy



dypy.eval("math.cos") #  at MEM_ADDRESS> (math is imported through get_value)

dypy.eval("math.cos", dynamic_args=True)(2, verbose=True) # 3, verbose is ignored (and math is imported through get_value)

dypy.eval("def my_function(x): return x + 1", function_of_interest="my_function") # 

dypy.eval("def my_function(x): return x + y", function_of_interest="my_function", context={"y": 2})(2) # 4

```



#### Variable Lookup



You can use `dypy.get_value` to lookup a variable as a string and then evaluate it. This is useful when you want to use a variable that is not defined in the current scope. You can do as follows:



```python

from dypy import get_value



get_value("math.pi") # 3.141592653589793

get_value("math.cos(0)") # won't evaluate, this is not a variable but a call to a variable

get_value("math.cos")(0) # 1.0



import math

get_value("cos", context=math) # math.cos



get_value("something_that_does_not_exist") # raises NameError

get_value("something_that_does_not_exist", strict=False) # None

```



`get_value` supports looking up variables in a module or package in your current working directory as well (as opposed to python's `import` which only supports looking up variables in the python standard library and installed packages). This is useful when you want to create a script that can be run from anywhere and still be able to access variables in the current working directory. 



For example, imagine you create your own python package with a runnable script that sorts files in a directory. You can use `get_value` to lookup a `config.sort` function in the current working directory. This way, you can create a `config.py` file in the current working directory and define your own sorting function. Or use a default sorting function if the `config.py` file does not exist. 



Your code would look like this:



```python

from dypy import get_value



def sort_files():

    sort_function = get_value("config.sort", strict=False) or default_sort

    # do something with sort

```



Although this example is somewhat contrived, such a use case is very common in data science and machine learning. Imagine writing a package for training a Deep Learning model. You can use `get_value` to lookup custom Dataset classes and model implementations and this way, people can use your package without the need to modifying your code, because they can simply inject their own implementations in places where you have used `get_value`.



Another potential use case is defining Neural Network layers. You can use `get_value` to lookup custom layers and use them in your model. Such as:



```python

from dypy import get_value

import torch



class MyLinearBlock(torch.nn.Module):

    def __init__(self, in_features, out_features, activation="torch.nn.ReLU"):

        super().__init__()

        self.linear = torch.nn.Linear(in_features, out_features)

        self.activation = get_value(activation, strict=False) 



    def forward(self, x):

        x = self.linear(x)

        x = self.activation(x) if self.activation else x

        return x

```



This way, you can change the activation function by simply changing the `activation` argument. For example, you can use `torch.nn.Sigmoid` or `torch.nn.Tanh` or even a custom activation function that you have defined in the current working directory or use one from a 3rd party package.



#### Dynamic Function Evaluation



What if you want to generate python programs dynamically? Meaning that you have a string that contains python code and you want to inject it into your program.  You can use `dypy.eval_function` to evaluate a piece of code and retrieve a function. You can do as follows:



```python

from dypy import eval_function



eval_function("lambda x: x + 1") #  at MEM_ADDRESS>



eval_function("def my_function(x): return x + 1") # wont work, this is not a function, 

# but a code block and you need to mention your function_of_interest in that code block



eval_function("def my_function(x): return x + 1", function_of_interest="my_function") # 

```



`eval_function` accepts three types of function descriptors:



1. A lambda function, e.g. `lambda x: x + 1`, or a code which evaluates to a callable object, `math.cos` (in this case, the values being looked up should be present in the evaluation context, more on this later).

2. A code block, which can contain multiple lines and functions, in which case you need to specify the name of the function of interest using the `function_of_interest` argument. `eval_function` will evaluate the code block and retrieve your function of interest.

3. A dictionary of "code", ["context", "function_of_interest"] pairs. Useful when you are using `eval_function` on top of a configuration file, in which case you can specify the code and the context for each function of interest.



When evaluating a function descriptor, you can specify a context in which the code will be evaluated. This is useful when you want to use variables that are not defined in the current scope. `dypy` has a context registry that you can use as a global context for all your function evaluations.



```python

from dypy import eval_function, register_context

import math



register_context(math, "math") # register math package as a context

# you can also use register(math), which will use the name of the package as the context name



eval_function("math.cos") #  at MEM_ADDRESS> (math is looked up through the context registry)

```



You can also specify a context for each function evaluation using the `context` argument. The context is a dictionary of variable names and their values. You can do as follows:



```python

eval_function("def my_function(x): return x + y", function_of_interest="my_function", context={"y": 1})(2) # 3

```



You can also optionally set `dynamic_args=True`, when evaluating a function. This will create a function that intelligently evaluates its arguments, by wrapping it using `dypy.dynamic_args_wrapper`. Functions wrapped by `dypy.dynamic_args_wrapper` preprocess arguments passed to them, and ignore arguments that are not defined in the function signature. For instance:



```python

eval_function("lambda x: x + 1", dynamic_args=True)(2, verbose=True) # 3, verbose is ignored

```



### Variable Assignment



There are times when you want to assign a variable in a dynamic manner. Meaning that you want to change a variable's value that is not necessarily defined in the current scope. You can use `dypy.set_value` to do so.  You can do as follows:



```python

from dypy import set_value



set_value("some_package.my_function", lambda x: x + 1)



# changing the value of pi in math package

set_value("math.pi", 3.14) 



# now if you import math, math.pi will be 3.14

import math

math.pi # 3.14

```



## License



dypy is licensed under the MIT License. See [LICENSE](LICENSE) for the full license text.



## Citation



If you use dypy in your research, please cite this repository as described in [CITATION.cff](CITATION.cff).