Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/apache/incubator-toree

Mirror of Apache Toree (Incubating)
https://github.com/apache/incubator-toree

toree

Last synced: 2 months ago
JSON representation

Mirror of Apache Toree (Incubating)

Lists

README 

        



[![Build Status][build-badge]][build-url]

[![License][license-badge]][license-url]

[![Join the chat at https://gitter.im/apache/toree][gitter-badge]][gitter-url]



[Apache Toree][website]

============

Apache Toree is a [Juypter Notebook](https://jupyter.org/) kernel. The main goal of Toree is to provide the foundation for

interactive applications that connect to and use [Apache Spark][1] using Scala language.



Overview

========

Toree provides an interface that allows clients to interact with a Spark Cluster. Clients can send libraries and

snippets of code that are interpreted and executed using a preconfigured Spark context.

These snippets can do a variety of things:

 1. Define and run spark jobs of all kinds

 2. Collect results from spark and push them to the client

 3. Load necessary dependencies for the running code

 4. Start and monitor a stream

 5. ...



Apache Toree supports the `Scala` programming language. It implements the latest Jupyter message protocol (5.0),

so it can easily plug into the latest releases of Jupyter/IPython (3.2.x+ and up) for quick, interactive data exploration.



Develop

=======

This project uses `make` as the entry point for build, test, and packaging. To perform a local build, you need to

install `sbt`, `jupyter/ipython`, and other development requirements locally on your machine.



To build and interact with Toree using Jupyter, run

```

make dev

```



This will start a Jupyter notebook server. Depending on your mode, it will be accessible at `http://localhost:8888` or `http://192.168.44.44:8888`. From here you can create notebooks that use Toree configured for Spark local mode.



Tests can be run by doing `make test`.



>> NOTE: Do not use `sbt` directly.



Build & Package

===============

To build and package up Toree, run

```

make release

```



This results in 2 packages.



- `./dist/toree--binary-release.tar.gz` is a simple package that contains JAR and executable

- `./dist/toree-.tar.gz` is a `pip` installable package that adds Toree as a Jupyter kernel.



NOTE: `make release` uses `docker`. Please refer to `docker` installation instructions for your system.



Run Examples

============

To play with the example notebooks, run

```

make jupyter

```



A notebook server will be launched in a `Docker` container with Toree and some other dependencies installed.

Refer to your `Docker` setup for the ip address. The notebook will be at `http://:8888/`.



Install

=======

This requires you to have a distribution of Apache Spark downloaded to the system where Apache Toree will run. The following commands will install Apache Toree.

```

pip install --upgrade toree

jupyter toree install --spark_home=

```

Dev snapshots of Toree are located at https://dist.apache.org/repos/dist/dev/incubator/toree. To install using one

of those packages, you can use the following:

```

pip install 

jupyter toree install --spark_home=

```

where `PIP_RELEASE_URL` is one of the `pip` packages. For example:



```

pip install https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshots/dev1/toree-pip/toree-0.2.0.dev1.tar.gz

jupyter toree install --spark_home=

```



Reporting Issues

================

Refer to and open issue [here][issues]



Communication

=============

You can reach us through [gitter][gitter-url] or our [mailing list][mail-list]



Version

=======

We are working on publishing binary releases of Toree soon. As part of our move into Apache Incubator, Toree will start a new version sequence starting at `0.1`.



Our goal is to keep `master` up to date with the latest version of Spark. When new versions of Spark require specific code changes to Toree, we will branch out older Spark version support.



As it stands, we maintain several branches for legacy versions of Spark. The table below shows what is available now.



Branch                       | Apache Spark Version

---------------------------- | --------------------

[master][master]             | 3.x.x

[0.4.x][0.4.x]               | 2.x.x

[0.1.x][0.1.x]               | 1.6+



Please note that for the most part, new features will mainly be added to the `master` branch.



Resources

=========



We are currently enhancing our documentation, which is available in our [website][documentation].



[1]: https://spark.apache.org/



[website]: https://toree.apache.org

[documentation]: https://toree.apache.org/docs/current/user/quick-start/

[issues]: https://issues.apache.org/jira/browse/TOREE

[build-badge]: https://github.com/apache/incubator-toree/actions/workflows/build.yml/badge.svg

[build-url]: https://github.com/apache/incubator-toree/actions/workflows/build.yml

[license-badge]: https://img.shields.io/badge/License-Apache%202-blue.svg?style=flat

[license-url]: LICENSE

[gitter-badge]: https://badges.gitter.im/Join%20Chat.svg

[gitter-url]: https://gitter.im/apache/toree

[try-jupyter]: https://try.jupyter.org

[mail-list]: mailto:[email protected]



[master]: https://github.com/apache/incubator-toree

[0.1.x]: https://github.com/apache/incubator-toree/tree/0.1.x

[0.4.x]: https://github.com/apache/incubator-toree/tree/0.4.x