https://github.com/hyzhak/decision-tree
Machine Learning: Decision Tree
https://github.com/hyzhak/decision-tree
classification data-science decision-tree machine-learning regression
Last synced: about 1 year ago
JSON representation
Machine Learning: Decision Tree
- Host: GitHub
- URL: https://github.com/hyzhak/decision-tree
- Owner: hyzhak
- License: mit
- Created: 2017-04-04T08:16:31.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2017-05-30T21:02:22.000Z (about 9 years ago)
- Last Synced: 2025-02-15T05:27:47.722Z (over 1 year ago)
- Topics: classification, data-science, decision-tree, machine-learning, regression
- Language: Jupyter Notebook
- Size: 173 KB
- Stars: 1
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Decision Tree
Machine Learning: Decision Tree. Function Approximation.
## Assumptions
Grow tree just big enough to fit correct labeled data.
So short decision tree is more preferable than long.
> Occam's razor: prefer the simplest hypothesis that fit the data
## Properties
- can approximate any discrete function
- we should should choose first parameter which reduce maximum of entropy (ID3)
```
I(X,Y) = H(X) - H(X|Y)
where:
H(X) = - sum(P(X=i) * log2(P(X=i))) for i in [1, n]
H(X|Y=v) = -sum(P(X=i|Y=v) * log2(P(X=i|Y=v)) for i in [1, n]
H(X|Y) = sum(P(Y=v) * H(X|Y=v)) for v in Y
```
We shouldn't use error rate criteria (what we care about) because
we can get stuck in local minimal
- C4.5
## Reduced-Error Pruning (Overfitting)
Split data into training and validation set
Create tree that classifies training set correctly
Do until further pruning is harmful:
1. Evaluate impact on validation set of pruning each possible node (plus those below it)
2. Greedily remove the one that most improves validation set accuracy
* produces smallest version of most accurate subtree
## Related
1. [No Free Lunch Theorem](https://en.wikipedia.org/wiki/No_free_lunch_theorem) *we can't really get a program to guess
a right function until it has seen all of examples*.
2. Kearns-Mansour’96 [On the Boosting Ability of Top-Down Decision Tree Learning Algorithms](https://www.cis.upenn.edu/~mkearns/papers/topdown.pdf)
# Link
1. [10-601, Spring 2015 by Tom Mitchell and Maria-Florina Balcan](http://www.cs.cmu.edu/~ninamf/courses/601sp15/lectures.shtml)