https://github.com/renardeinside/nocturne

Useful elements and building blocks for scalable Deep Learning applications on Databricks.
https://github.com/renardeinside/nocturne

databricks deep-learning gpu horovod petastorm spark

Last synced: 9 days ago
JSON representation

Useful elements and building blocks for scalable Deep Learning applications on Databricks.

• Host: GitHub
• URL: https://github.com/renardeinside/nocturne
• Owner: renardeinside
• Created: 2022-03-04T10:49:15.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2022-04-04T14:40:21.000Z (almost 3 years ago)
• Last Synced: 2024-12-12T16:13:39.210Z (2 months ago)
• Topics: databricks, deep-learning, gpu, horovod, petastorm, spark
• Language: Python
• Homepage:
• Size: 686 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst

Awesome Lists containing this project

README

        
Nocturne - utilities for Scalable Deep Learning with Spark, Horovod and Petastorm

=================================================================================

This repository contains useful elements and building blocks for scalable Deep Learning applications.

|build| |codecov| |black| |pypi|

.. |build| image:: https://img.shields.io/github/workflow/status/renardeinside/nocturne/CI%20pipeline/main?style=for-the-badge

    :alt: GitHub Workflow Status

    :target: https://github.com/renardeinside/dbx_scalable_dl/actions/workflows/onpush.yml

.. |pypi| image:: https://img.shields.io/pypi/v/nocturne.svg?style=for-the-badge

    :target: https://pypi.org/project/nocturne/

    :alt: Latest Python Release

.. |black| image:: https://img.shields.io/badge/code%20style-black-000000.svg?style=for-the-badge

    :target: https://github.com/psf/black

    :alt: We use black for formatting

.. |codecov| image:: https://img.shields.io/codecov/c/github/renardeinside/nocturne/main?style=for-the-badge&token=P9CiNFvruh

    :alt: Codecov branch

    :target: https://app.codecov.io/gh/renardeinside/nocturne

Dependencies

------------

The following libraries shall be installed before using :code:`nocturne`:

* JDK 1.8

* Apache Spark 3.x

* Tensorflow 2.x

* :code:`horovod[spark, tensorflow]`

* :code:`petastorm`

Since packaging of these dependencies might be challenging, you can use the base docker image with all dependencies provided in `Dockerfile.base`_.

To install the library, run:

.. code-block::

    pip install nocturne

Resources

---------

* `Horovod installation guide `_

.. _Dockerfile.base: docker/Dockerfile.base