Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/openscilab/pymilo
PyMilo: Python for ML I/O
https://github.com/openscilab/pymilo
artificial-intelligence artificial-intelligence-algorithms artificial-neural-networks deep-learning deserialization devops devops-tools machine-learning machine-learning-algorithms ml ml-models-export ml-models-import mlops scikit-learn serialization
Last synced: about 1 month ago
JSON representation
PyMilo: Python for ML I/O
- Host: GitHub
- URL: https://github.com/openscilab/pymilo
- Owner: openscilab
- License: mit
- Created: 2022-12-08T22:49:33.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-24T08:12:53.000Z (8 months ago)
- Last Synced: 2024-04-24T09:25:54.380Z (8 months ago)
- Topics: artificial-intelligence, artificial-intelligence-algorithms, artificial-neural-networks, deep-learning, deserialization, devops, devops-tools, machine-learning, machine-learning-algorithms, ml, ml-models-export, ml-models-import, mlops, scikit-learn, serialization
- Language: Python
- Homepage:
- Size: 743 KB
- Stars: 122
- Watchers: 4
- Forks: 5
- Open Issues: 21
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Support: SUPPORTED_MODELS.md
- Authors: AUTHORS.md
Awesome Lists containing this project
README
----------
## Overview
PyMilo is an open source Python package that provides a simple, efficient, and safe way for users to export pre-trained machine learning models in a transparent way. By this, the exported model can be used in other environments, transferred across different platforms, and shared with others. PyMilo allows the users to export the models that are trained using popular Python libraries like scikit-learn, and then use them in deployment environments, or share them without exposing the underlying code or dependencies. The transparency of the exported models ensures reliability and safety for the end users, as it eliminates the risks of binary or pickle formats.
PyPI Counter
Github Stars
Branch
main
dev
CI
Code Quality
## Installation
### PyPI
- Check [Python Packaging User Guide](https://packaging.python.org/installing/)
- Run `pip install pymilo==1.0`
### Source code
- Download [Version 1.0](https://github.com/openscilab/pymilo/archive/v1.0.zip) or [Latest Source](https://github.com/openscilab/pymilo/archive/dev.zip)
- Run `pip install .`
### Conda
- Check [Conda Managing Package](https://conda.io/)
- Update Conda using `conda update conda`
- Run `conda install -c openscilab pymilo`
## Usage
### Import/Export
Imagine you want to train a `LinearRegression` model representing this equation: $y = x_0 + 2x_1 + 3$. You will create data points (`X`, `y`) and train your model as follows.
```pycon
>>> import numpy as np
>>> from sklearn.linear_model import LinearRegression
>>> X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
>>> y = np.dot(X, np.array([1, 2])) + 3
# y = 1 * x_0 + 2 * x_1 + 3
>>> model = LinearRegression().fit(X, y)
>>> pred = model.predict(np.array([[3, 5]]))
# pred = [16.] (=1 * 3 + 2 * 5 + 3)
```
Using PyMilo `Export` class you can easily serialize and export your trained model into a JSON file.
```pycon
>>> from pymilo import Export
>>> Export(model).save("model.json")
```
You can check out your model as a JSON file now.
```json
{
"data": {
"fit_intercept": true,
"copy_X": true,
"n_jobs": null,
"positive": false,
"n_features_in_": 2,
"coef_": {
"pymiloed-ndarray-list": [
1.0000000000000002,
1.9999999999999991
],
"pymiloed-ndarray-dtype": "float64",
"pymiloed-ndarray-shape": [
2
],
"pymiloed-data-structure": "numpy.ndarray"
},
"rank_": 2,
"singular_": {
"pymiloed-ndarray-list": [
1.618033988749895,
0.6180339887498948
],
"pymiloed-ndarray-dtype": "float64",
"pymiloed-ndarray-shape": [
2
],
"pymiloed-data-structure": "numpy.ndarray"
},
"intercept_": {
"value": 3.0000000000000018,
"np-type": "numpy.float64"
}
},
"sklearn_version": "1.4.2",
"pymilo_version": "0.8",
"model_type": "LinearRegression"
}
```
You can see all the learned parameters of the model in this file and change them if you want. This JSON representation is a transparent version of your model.
Now let's load it back. You can do it easily by using PyMilo `Import` class.
```pycon
>>> from pymilo import Import
>>> model = Import("model.json").to_model()
>>> pred = model.predict(np.array([[3, 5]]))
# pred = [16.] (=1 * 3 + 2 * 5 + 3)
```
This loaded model is exactly the same as the original trained model.
### ML streaming
You can easily serve your ML model from a remote server using `ML streaming` feature of PyMilo.
⚠️ `ML streaming` feature exists in versions `>=1.0`
⚠️ In order to use `ML streaming` feature, make sure you've installed the `streaming` mode of PyMilo
#### Server
Let's assume you are in the remote server and you want to import the exported JSON file and start serving your model!
```pycon
>>> from pymilo import Import
>>> from pymilo.streaming import PymiloServer
>>> my_model = Import("model.json").to_model()
>>> communicator = PymiloServer(model=my_model, port=8000).communicator
>>> communicator.run()
```
Now `PymiloServer` runs on port `8000` and exposes REST API to `upload`, `download` and retrieve **attributes** either **data attributes** like `model._coef` or **method attributes** like `model.predict(x_test)`.
#### Client
By using `PymiloClient` you can easily connect to the remote `PymiloServer` and execute any functionalities that the given ML model has, let's say you want to run `predict` function on your remote ML model and get the result:
```pycon
>>> from pymilo.streaming import PymiloClient
>>> pymilo_client = PymiloClient(mode=PymiloClient.Mode.LOCAL, server_url="SERVER_URL")
>>> pymilo_client.toggle_mode(PymiloClient.Mode.DELEGATE)
>>> result = pymilo_client.predict(x_test)
```
ℹ️ If you've deployed `PymiloServer` locally (on port `8000` for instance), then `SERVER_URL` would be `http://127.0.0.1:8000`
You can also download the remote ML model into your local and execute functions locally on your model.
Calling `download` function on `PymiloClient` will sync the local model that `PymiloClient` wraps upon with the remote ML model, and it doesn't save model directly to a file.
```pycon
>>> pymilo_client.download()
```
If you want to save the ML model to a file in your local, you can use `Export` class.
```pycon
>>> from pymilo import Export
>>> Export(pymilo_client.model).save("model.json")
```
Now that you've synced the remote model with your local model, you can run functions.
```pycon
>>> pymilo_client.toggle_mode(mode=PymiloClient.Mode.LOCAL)
>>> result = pymilo_client.predict(x_test)
```
`PymiloClient` wraps around the ML model, either to the local ML model or the remote ML model, and you can work with `PymiloClient` in the exact same way that you did with the ML model, you can run exact same functions with same signature.
ℹ️ Through the usage of `toggle_mode` function you can specify whether `PymiloClient` applies requests on the local ML model `pymilo_client.toggle_mode(mode=Mode.LOCAL)` or delegates it to the remote server `pymilo_client.toggle_mode(mode=Mode.DELEGATE)`
## Supported ML models
| scikit-learn | PyTorch |
| ---------------- | ---------------- |
| Linear Models ✅ | - |
| Neural Networks ✅ | - |
| Trees ✅ | - |
| Clustering ✅ | - |
| Naïve Bayes ✅ | - |
| Support Vector Machines (SVMs) ✅ | - |
| Nearest Neighbors ✅ | - |
| Ensemble Models ✅ | - |
| Pipeline Model ✅ | - |
| Preprocessing Models ✅ | - |
Details are available in [Supported Models](https://github.com/openscilab/pymilo/blob/main/SUPPORTED_MODELS.md).
## Issues & bug reports
Just fill an issue and describe it. We'll check it ASAP! or send an email to [[email protected]](mailto:[email protected] "[email protected]").
- Please complete the issue template
You can also join our discord server
## Acknowledgments
[Python Software Foundation (PSF)](https://www.python.org/psf/) grants PyMilo library partially for version **1.0**. [PSF](https://www.python.org/psf/) is the organization behind Python. Their mission is to promote, protect, and advance the Python programming language and to support and facilitate the growth of a diverse and international community of Python programmers.
[Trelis Research](https://trelis.com/) grants PyMilo library partially for version **1.0**. [Trelis Research](https://trelis.com/) provides tools and tutorials for businesses and developers looking to fine-tune and deploy large language models.
## Show your support
### Star this repo
Give a ⭐️ if this project helped you!
### Donate to our project
If you do like our project and we hope that you do, can you please support us? Our project is not and is never going to be working for profit. We need the money just so we can continue doing what we do ;-) .
