https://github.com/saurfang/spark-tsne

Distributed t-SNE via Apache Spark
https://github.com/saurfang/spark-tsne

spark tsne

Last synced: 3 months ago
JSON representation

Distributed t-SNE via Apache Spark

• Host: GitHub
• URL: https://github.com/saurfang/spark-tsne
• Owner: saurfang
• License: apache-2.0
• Created: 2015-07-06T19:01:28.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2017-12-09T05:19:53.000Z (about 7 years ago)
• Last Synced: 2024-10-31T21:35:14.266Z (3 months ago)
• Topics: spark, tsne
• Language: Scala
• Homepage: https://saurfang.github.io/spark-tsne-demo/tsne-pixi.html
• Size: 96.7 KB
• Stars: 161
• Watchers: 13
• Forks: 37
• Open Issues: 7
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# spark-tsne

[![Join the chat at https://gitter.im/saurfang/spark-tsne](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/saurfang/spark-tsne?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Build Status](https://travis-ci.org/erwinvaneijk/spark-tsne.svg?branch=master)](https://travis-ci.org/erwinvaneijk/spark-tsne)

Distributed [t-SNE](http://lvdmaaten.github.io/tsne/) with Apache Spark. WIP...

t-SNE is a dimension reduction technique that is particularly good for visualizing high

dimensional data. This is an attempt to implement this algorithm using Spark to leverage

distributed computing power.

The project is still in progress of replicating reference implementations from the original

papers. Spark specific optimizations will be the next goal once the correctness is verified.

Currently I'm showcasing this using the standard [MNIST](http://yann.lecun.com/exdb/mnist/)

handwriting recognition dataset. I have created a [WebGL player](https://saurfang.github.io/spark-tsne-demo/tsne-pixi.html)

(built using [pixi.js](https://github.com/pixijs/pixi.js)) to visualize the inner workings

as well as the final results of t-SNE. If a WebGL is unavailable for you, you may checkout

the [d3.js player](https://saurfang.github.io/spark-tsne-demo/tsne.html) instead.

![](data/mnist/tsne.gif)

## Credits

- [t-SNE Julia implementation](https://github.com/lejon/TSne.jl)

- [Barnes-Hut t-SNE](https://github.com/lvdmaaten/bhtsne/)