Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/altescy/metamaker
⚗️ Simple command line tool to train and deploy your machine learning models with AWS SageMaker
https://github.com/altescy/metamaker
fastapi machine-learning poetry python sagemaker
Last synced: 2 months ago
JSON representation
⚗️ Simple command line tool to train and deploy your machine learning models with AWS SageMaker
- Host: GitHub
- URL: https://github.com/altescy/metamaker
- Owner: altescy
- License: mit
- Created: 2021-11-05T12:25:29.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-02-08T02:17:16.000Z (almost 2 years ago)
- Last Synced: 2024-11-15T04:48:57.763Z (3 months ago)
- Topics: fastapi, machine-learning, poetry, python, sagemaker
- Language: Python
- Homepage: https://pypi.org/project/metamaker/
- Size: 448 KB
- Stars: 5
- Watchers: 3
- Forks: 1
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
metamaker
=========
[](https://github.com/altescy/metamaker/actions/workflows/main.yaml)
[](https://github.com/altescy/metamaker/blob/master/LICENSE)
[](https://github.com/altescy/metamaker)
[](https://pypi.org/project/metamaker/)
Simple command line tool to train and deploy your machine learning models with AWS SageMaker
## Features
metamaker enables you to:
- Build a docker image for training and inference with [poetry](https://python-poetry.org/) and [FastAPI](https://fastapi.tiangolo.com/)
- Train your own machine learning model with SageMaker
- Deploy inference endpoint with SageMaker
## Usage
1. Create poetry project and install metamaker
```
❯ poetry new your_module
❯ cd your_module
❯ poetry add metamaker
```
2. Define scripts for traning and inference in `main.py`
```main.py
from pathlib import Path
from typing import Any, Dict
from metamaker import MetaMaker
# Import your model, and input/output data classs:
#
# Model ... machine learning model class you want to use
# Input ... input data class for inference
# Output ... ouput data class for inference
#
# Note that the Input and Output are used as type hints to
# create API endpoint with FastAPI like below:
#
# @fastapi_app.post("/invocations")
# def predict(data: Input) -> Output:
# ...
from your_module import Model, Input, Output
app = MetaMaker[Model, Input, Output]()
@app.trainer
def train(
dataset_path: Path,
artifact_path: Path,
hyperparameters: Dict[str, Any],
) -> None:
model = Model(**hyperparameters)
model.train(dataset_path / "train.csv")
model.save(artifact_path / "model.tar.gz")
@app.loader
def load(artifact_path: Path) -> Model:
return Model.load(artifact_path / "model.tar.gz")
@app.predictor
def predict(model: Model, data: Input) -> Output:
return model.predict(data)
```
3. Write metamaker configs in `metamaker.yaml`
```metamaker.yaml
# Specify metamaker handler like: `path.to.module:app_name`
handler: main:app
# dataset_path and artifact_path should be directories and end with '/'
dataset_path: s3://your-bucket/path/to/dataset/
artifact_path: s3://your-bucket/path/to/artifacts/
hyperparameter_path: ./hparams.yaml
image:
name: metamaker
includes:
- your_module/
- main.py
excludes:
- __pycache__/
- '*.py[cod]'
training:
execution_role: arn:aws:iam::xxxxxxxxxxxx:role/SageMakerExecutionRole
instance:
type: ml.m5.large
count: 1
inference:
endpoint_name: your_endpoint
instance:
type: ml.t2.meduim
count: 1
```
4. Build docker image and push to ECR
```
metamaker build --deploy .
```
5. Train your model with SageMaker and deploy endpoint
```
metamaker sagemaker train --deploy
```