{"id":13701597,"url":"https://github.com/better/crossfader","last_synced_at":"2025-04-10T19:30:54.211Z","repository":{"id":27607802,"uuid":"31091440","full_name":"better/crossfader","owner":"better","description":"Autoencoders to find structure in arbitrary datasets","archived":false,"fork":false,"pushed_at":"2015-10-28T13:21:11.000Z","size":29726,"stargazers_count":122,"open_issues_count":0,"forks_count":11,"subscribers_count":138,"default_branch":"master","last_synced_at":"2025-03-25T03:02:00.942Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/better.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-02-21T00:07:25.000Z","updated_at":"2025-03-24T04:37:47.000Z","dependencies_parsed_at":"2022-08-07T13:00:41.892Z","dependency_job_id":null,"html_url":"https://github.com/better/crossfader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/better%2Fcrossfader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/better%2Fcrossfader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/better%2Fcrossfader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/better%2Fcrossfader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/better","download_url":"https://codeload.github.com/better/crossfader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248281395,"owners_count":21077423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T20:01:51.138Z","updated_at":"2025-04-10T19:30:52.753Z","avatar_url":"https://github.com/better.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":".. figure:: https://rawgit.com/better/crossfader/master/demo.png\n   :align: center\n\nWhat is this?\n-------------\n\nAn experimental model for robust dimensionality reduction of arbitrary data sets.\nIt lets you explore the marginal distributions of all parameters with any parameter(s) fixed.\nIt is also extremely fast at computing these probability distributions.\nIt assumes nothing about the distribution of the features, which can have any units and have any scale.\n\nWhat can it be used for?\n------------------------\n\n* Predicting missing values\n* Visualizing probability distributions\n* Finding outliers\n* Modelling uncertainties\n\nDemo Time\n---------\n\n`Here is a demo \u003chttps://rawgit.com/better/crossfader/master/demo.html#wine\u003e`_ trained on a bunch of different data sets.\nThe demo is written in JS and uses pre-trained models.\n\nMore background\n---------------\n\nTools like `Crossfilter \u003chttp://square.github.io/crossfilter/\u003e`_ are great at visualizing datasets and how features are correlated.\nCrossfilter renders real data points based on a number of feature selections.\nHowever as the number of features increase, it gets harder to find data points that fulfill all criteria.\nThis is the `curse of dimensionality \u003chttp://en.wikipedia.org/wiki/Curse_of_dimensionality\u003e`_ problem which often makes analysis of high-dimensional data hard.\n\nThis package has a different approach.\nIt computes a statistical model of the underlying data.\nThe downside is that you can no longer explore real data points.\nThe upside is you can explore conditional dependencies and make predictions about data you have not observed.\n\nHow does it work?\n-----------------\n\nIt builds an `autoencoder \u003chttp://en.wikipedia.org/wiki/Autoencoder\u003e`_ that learns to reconstruct missing data.\n\nTo be able to work with any distributions, it reduces all inputs to a series of binary values.\nEvery feature is encoded as a binary feature vector by constructing splits from the empirical distribution of the training data.\nThe predicted probabilities for each split then gives the `CDF \u003chttp://en.wikipedia.org/wiki/Cumulative_distribution_function\u003e`_ directly.\n\nThe autoencoder has a series of hidden bottleneck layers (typically 2-5 layers with 20-100 units).\nOne way to think of it is that the autoencoder finds a low-dimensional manifold in the high-dimensional space.\nThis manifold can be highly nonlinear due to the nonlinearities in the autoencoder.\nThe autoencoder then essentially learns a projection from the high dimensional space onto the manifold and another projection back to the original space.\nThe dimensionality reduction is effectively a way of getting around the curse of dimensionality.\n\nThe autoencoder is trained using `Theano \u003chttp://deeplearning.net/software/theano/\u003e`_ in Python.\nYou can run it on a GPU although the speed improvements are not drastic because of some bottlenecks.\n\nLimitations and future work\n---------------------------\n\n* Training the autoencoder is very slow. It can take a few hours for small data sets (with 10-100 features).\n* The training time grows with the number of features.\n* The training time should stay relatively constant with the size of the training data since it is using `stochastic gradient descent \u003chttp://en.wikipedia.org/wiki/Stochastic_gradient_descent\u003e`_.\n* It only handles numerical attributes, although categorical should be pretty easy to add.\n* The model probably overfits quite a bit and should do proper cross-validation to measure this.\n* Similarly there is a bunch of hyperparameters that have to be tuned to your data set.\n\nLicense\n-------\n\nReleased under the Apache 2.0 license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbetter%2Fcrossfader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbetter%2Fcrossfader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbetter%2Fcrossfader/lists"}