Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/Ibotta/sk-dist

Distributed scikit-learn meta-estimators in PySpark
https://github.com/Ibotta/sk-dist

data-science machine-learning ml scikit-learn spark

Last synced: 3 months ago
JSON representation

Distributed scikit-learn meta-estimators in PySpark

Awesome Lists containing this project

README 

        
.. figure:: https://github.com/Ibotta/sk-dist/blob/master/doc/images/skdist.png

   :alt: sk-dist



sk-dist: Distributed scikit-learn meta-estimators in PySpark

============================================================



|License| |PyPI Package| |Downloads| |Python Versions|



What is it?

-----------



``sk-dist`` is a Python package for machine learning built on top of

`scikit-learn `__ and is

distributed under the `Apache 2.0 software

license `__. The

``sk-dist`` module can be thought of as "distributed scikit-learn" as

its core functionality is to extend the ``scikit-learn`` built-in

``joblib`` parallelization of meta-estimator training to

`spark `__. A popular use case is the 

parallelization of grid search as shown here:



.. figure:: https://github.com/Ibotta/sk-dist/blob/master/doc/images/grid_search.png

   :alt: sk-dist



Check out the `blog post `__ 

for more information on the motivation and use cases of ``sk-dist``.



Main Features

-------------



-  **Distributed Training** - ``sk-dist`` parallelizes the training of

   ``scikit-learn`` meta-estimators with PySpark. This allows

   distributed training of these estimators without any constraint on

   the physical resources of any one machine. In all cases, spark

   artifacts are automatically stripped from the fitted estimator. These

   estimators can then be pickled and un-pickled for prediction tasks,

   operating identically at predict time to their ``scikit-learn``

   counterparts. Supported tasks are:



   -  *Grid Search*: `Hyperparameter optimization

      techniques `__,

      particularly

      `GridSearchCV `__

      and

      `RandomizedSeachCV `__,

      are distributed such that each parameter set candidate is trained

      in parallel.

   -  *Multiclass Strategies*: `Multiclass classification

      strategies `__,

      particularly

      `OneVsRestClassifier `__

      and

      `OneVsOneClassifier `__,

      are distributed such that each binary probelm is trained in

      parallel.

   -  *Tree Ensembles*: `Decision tree

      ensembles `__

      for classification and regression, particularly

      `RandomForest `__

      and

      `ExtraTrees `__,

      are distributed such that each tree is trained in parallel.



-  **Distributed Prediction** - ``sk-dist`` provides a prediction module

   which builds `vectorized

   UDFs `__

   for

   `PySpark `__

   `DataFrames `__

   using fitted ``scikit-learn`` estimators. This distributes the

   ``predict`` and ``predict_proba`` methods of ``scikit-learn``

   estimators, enabling large scale prediction with ``scikit-learn``.

-  **Feature Encoding** - ``sk-dist`` provides a flexible feature

   encoding utility called ``Encoderizer`` which encodes mix-typed

   feature spaces using either default behavior or user defined

   customizable settings. It is particularly aimed at text features, but

   it additionally handles numeric and dictionary type feature spaces.



Installation

------------



Dependencies

~~~~~~~~~~~~



``sk-dist`` requires:



-  `Python `__ (>= 3.5)

-  `scikit-learn `__ (>=0.20.0,<0.23.2)

-  `pandas `__ (>=0.17.0)

-  `numpy `__ 

-  `scipy `__ 

-  `joblib `__ 



Dependency Notes

~~~~~~~~~~~~~~~~



-  versions of ``numpy``, ``scipy`` and ``joblib`` that are compatible with any supported version of ``scikit-learn`` should be sufficient for ``sk-dist``

- ``sk-dist`` is not supported with Python 2



Spark Dependencies

~~~~~~~~~~~~~~~~~~



Most ``sk-dist`` functionality requires a spark installation as well as

PySpark. Some functionality can run without spark, so spark related

dependencies are not required. The connection between sk-dist and spark

relies solely on a ``sparkContext`` as an argument to various

``sk-dist`` classes upon instantiation.



A variety of spark configurations and setups will work. It is left up to

the user to configure their own spark setup. The testing suite runs

``spark 2.4`` and ``spark 3.0``, though any ``spark 2.0+`` versions 

are expected to work.



Additional spark related dependecies are ``pyarrow``, which is used only

for ``skdist.predict`` functions. This uses vectorized pandas UDFs which

require ``pyarrow>=0.8.0``, tested with ``pyarrow==0.16.0``. 

Depending on the spark version, it may be necessary to set

``spark.conf.set("spark.sql.execution.arrow.enabled", "true")`` in the

spark configuration.



User Installation

~~~~~~~~~~~~~~~~~



The easiest way to install ``sk-dist`` is with ``pip``:



::



    pip install --upgrade sk-dist



You can also download the source code:



::



    git clone https://github.com/Ibotta/sk-dist.git



Testing

~~~~~~~



With ``pytest`` installed, you can run tests locally:



::



    pytest sk-dist



Examples

--------



The package contains numerous 

`examples `__ 

on how to use ``sk-dist`` in practice. Examples of note are:



-  `Grid Search with XGBoost `__

-  `Spark ML Benchmark Comparison `__

-  `Encoderizer with 20 Newsgroups `__

-  `One-Vs-Rest vs One-Vs-One `__

-  `Large Scale Sklearn Prediction with PySpark UDFs `_



Gradient Boosting

-----------------



``sk-dist`` has been tested with a number of popular gradient boosting packages that conform to the ``scikit-learn`` API. This 

includes ``xgboost`` and ``catboost``. These will need to be installed in addition to ``sk-dist`` on all nodes of the spark 

cluster via a node bootstrap script. Version compatibility is left up to the user.



Support for ``lightgbm`` is not guaranteed, as it requires `additional installations `__ on all 

nodes of the spark cluster. This may work given proper installation but has not beed tested with ``sk-dist``.



Background

----------



The project was started at `Ibotta

Inc. `__ on the machine learning

team and open sourced in 2019.



It is currently maintained by the machine learning team at Ibotta. Special

thanks to those who contributed to ``sk-dist`` while it was initially

in development at Ibotta:



-  `Evan Harris `__

-  `Nicole Woytarowicz `__

-  `Mike Lewis `__

-  `Bobby Crimi `__



Thanks to `James Foley `__ for logo artwork.



.. figure:: https://github.com/Ibotta/sk-dist/blob/master/doc/images/ibottaml.png

   :alt: IbottaML



.. |License| image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg

   :target: https://opensource.org/licenses/Apache-2.0

.. |Build Status| image:: https://travis-ci.com/Ibotta/sk-dist.png?branch=master

   :target: https://travis-ci.com/Ibotta/sk-dist

.. |PyPI Package| image:: https://badge.fury.io/py/sk-dist.svg

   :target: https://pypi.org/project/sk-dist/

.. |Downloads| image:: https://pepy.tech/badge/sk-dist

   :target: https://pepy.tech/project/sk-dist

.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/sk-dist

   :target: https://pypi.org/project/sk-dist/