https://github.com/csinva/trees-to-networks

Bridging random forests and deep neural networks. Partial implementation of "Neural Random Forests" https://arxiv.org/abs/1604.07143
https://github.com/csinva/trees-to-networks

artificial-intelligence classification decision-tree decision-tree-classifier deep-learning machine-learning machinelearning neural-network neural-networks paper-implementations python pytorch random-forest scikit-learn statistics

Last synced: about 2 months ago
JSON representation

Bridging random forests and deep neural networks. Partial implementation of "Neural Random Forests" https://arxiv.org/abs/1604.07143

• Host: GitHub
• URL: https://github.com/csinva/trees-to-networks
• Owner: csinva
• Created: 2019-06-28T19:23:51.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2019-07-04T15:04:18.000Z (almost 7 years ago)
• Last Synced: 2025-02-08T13:42:51.781Z (over 1 year ago)
• Topics: artificial-intelligence, classification, decision-tree, decision-tree-classifier, deep-learning, machine-learning, machinelearning, neural-network, neural-networks, paper-implementations, python, pytorch, random-forest, scikit-learn, statistics
• Language: Jupyter Notebook
• Homepage:
• Size: 429 KB
• Stars: 3
• Watchers: 4
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

          
## Bridging trees and neural networks.

**Code here converts sklearn tree -> pytorch DNN**

`rf_to_dnn_ex.ipynb` shows a simple example of converting a random forest to a DNN

Both random forests and DNNs are very strong predictive models. This [cool recent paper title "Neural Random Forests"](https://arxiv.org/pdf/1604.07143.pdf) gives a simple algorithm for exactly rewriting any random forest as a sparse neural network. This could be useful then for combining the inductive biases of both and other interesting things.

## how does it work?

![](biau_19_fig2.png)

The idea is to rewrite the neural network into 3 layers:

1. The first layer identifies whether a point is on the left or right side of a split (outputs -1, 1)

   - $out_{split} = \text{sign}(in - thresh)$

   - this can be made faster by doing indexing rather than a matrix multiply (maybe sparse tensor will be fast enough)

2. The second layer determines whether a a point is in a leaf or not (ouputs 0/1)

	- $out_{leaf} = (\sum w \cdot in) ==  depth(leaf)$

3. the final layer simply multiplies the vector of (0s/1s) by the value of the leaf

  - $out_{pred} = \sum_{leaf} out_{leaf} \cdot val_{leaf}$

  - the 0-1 helps make it sparser (and simpler :smile:)