Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/piotrlaczkowski/keras-data-processor
Data Preprocessing model based on Keras preprocessing layers that can be used as a standalone model or incorporated to Keras model as first layers.
https://github.com/piotrlaczkowski/keras-data-processor
data keras layers preprocessing tensorflow
Last synced: 28 days ago
JSON representation
Data Preprocessing model based on Keras preprocessing layers that can be used as a standalone model or incorporated to Keras model as first layers.
- Host: GitHub
- URL: https://github.com/piotrlaczkowski/keras-data-processor
- Owner: piotrlaczkowski
- License: mit
- Created: 2024-03-08T19:22:05.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-04-12T11:26:55.000Z (7 months ago)
- Last Synced: 2024-04-12T14:47:25.843Z (7 months ago)
- Topics: data, keras, layers, preprocessing, tensorflow
- Language: Python
- Homepage: https://piotrlaczkowski.github.io/keras-data-processor/
- Size: 1.94 MB
- Stars: 1
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: docs/contributing.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# 🌟 Welcome to Keras Data Processor (KDP) - Preprocessing Power with TensorFlow Keras 🌟
**Welcome to the Future of Data Preprocessing!**
Diving into the world of machine learning and data science, we often find ourselves tangled in the preprocessing jungle.
Worry no more! Introducing a state-of-the-art data preprocessing model based on TensorFlow Keras and the innovative use of Keras preprocessing layers.
Say goodbye to tedious data preparation tasks and hello to streamlined, efficient, and scalable data pipelines. Whether you're a seasoned data scientist or just starting out, this tool is designed to supercharge your ML workflows, making them more robust and faster than ever!
## 🔑 Key Features:
- Automatic and scalable features statistics extraction: Automatically infer the feature tatistics from your data, saving you time and efforts.
- Customizable Preprocessing Pipelines: Tailor your preprocessing steps with ease, choosing from a wide range of options for numeric, categorical, and even complex feature crosses.
- Scalability and Efficiency: Designed for performance, handling large datasets with ease thanks to TensorFlow's powerful backend.
- Easy Integration: Seamlessly fits into your TensorFlow Keras models (as first layers of the mode), making it a breeze to go from raw data to trained model faster than ever.
## 🚀 Getting started:
We use poetry for handling dependencies so you will need to install it first.
Then you can install the dependencies by running:
To install dependencies:
```bash
poetry install
```
or to enter a dedicated env directly:
```bash
poetry shell
```
Then you can simply configure your preprocessor:
## 🛠️ Building Preprocessor:
```python
from kdp import PreprocessingModel
from kdp import FeatureType
# DEFINING FEATURES PROCESSORS
features_specs = {
# ======= NUMERICAL Features =========================
"feat1": FeatureType.FLOAT_NORMALIZED,
"feat2": FeatureType.FLOAT_RESCALED,
# ======= CATEGORICAL Features ========================
"feat3": FeatureType.STRING_CATEGORICAL,
"feat4": FeatureType.INTEGER_CATEGORICAL,
# ======= TEXT Features ========================
"feat5": FeatureType.TEXT,
}
# INSTANTIATE THE PREPROCESSING MODEL with your data
ppr = PreprocessingModel(
path_data="data/my_data.csv",
features_specs=features_spec,
)
# construct the preprocessing pipelines
ppr.build_preprocessor()
```
This wil output:
```JS
{
'model': ,
'inputs': {
'feat1': ,
'feat2': ,
'feat3': ,
'feat4': ,
'feat5':
},
'signature': {
'feat1': TensorSpec(shape=(None, 1), dtype=tf.float32, name='feat1'),
'feat2': TensorSpec(shape=(None, 1), dtype=tf.float32, name='feat2'),
'feat3': TensorSpec(shape=(None, 1), dtype=tf.string, name='feat3'),
'feat4': TensorSpec(shape=(None, 1), dtype=tf.int32, name='feat4'),
'feat5': TensorSpec(shape=(None, 1), dtype=tf.string, name='feat5')
},
'output_dims': 45
}
```
This will result in the following preprocessing steps:
**This preprocessing model can be used independentyly or as the first layer of any Keras model.
This means you can ship your model with the preprocessing pipeline (built-in) as a single entity and deploy it with ease using Tesnorflow Serving.**
```python
## 🔍 Dive Deeper:
Explore the detailed documentation to leverage the full potential of this preprocessing tool. Learn about customizing feature crosses, bucketization strategies, embedding sizes, and much more to truly tailor your preprocessing pipeline to your project's needs.