https://github.com/pabsan-0/vfs2
Vectorial Mutual-Information based feature selection
https://github.com/pabsan-0/vfs2
feature-selection mutual-information repos-ml statistics
Last synced: over 1 year ago
JSON representation
Vectorial Mutual-Information based feature selection
- Host: GitHub
- URL: https://github.com/pabsan-0/vfs2
- Owner: pabsan-0
- Created: 2022-05-21T17:40:46.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2022-06-27T11:16:16.000Z (almost 4 years ago)
- Last Synced: 2025-01-23T16:53:04.237Z (over 1 year ago)
- Topics: feature-selection, mutual-information, repos-ml, statistics
- Language: Python
- Homepage:
- Size: 78.1 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Vector feature selection
Mutual-information based feature selection considering feature sets rather than single-dimensional features.
The mutual information (MI) among two random variables X and Y, I(X;Y) can be computed from their joint and marginal probability density functions (pdf) *fxy*, *fx* and *fy*. The MI can be expanded to random vectors ***X*** and ***Y***, however their pdf estimation becomes much harder.
Take the three types of feature selection methods:
- Forward selection
- Backward elimination
- Exhaustive search
Mutual information based selection methods in the literature traditionally follow the Forward Selection approach by using a variety of scores such as:
- MIM
- MRMR
- JMIM
- etc.
Which consist of different combinations of low-dim MI among the different candidate features and the target, keeping to the trivariate case MI(X,Y;Z) at most and avoiding the hindrance of estimating high-dimensional probability densities.
This repository provides implementations for:
- Mutual Information MI(***X***;***Y***)
- Forward selection methods in the literature
- Backward elimination from the methods in the literature
- Exhaustive selection based on the raw MI(***X***;***Y***)
- Vectorized versions of MRMR and DISR methods by replacing f(MI(X*i*;Y)) -> MI(***X***;***Y***)
:boom: See next section [Shorts](#shorts) for a super brief tutorial on how to use the library.
:boom: See subdirectory [scripts](scripts) for more detailed examples.
### Shorts
```
from vfs import *
from vfs.shorthands import df_iris, MRMR
df, features, targets = df_iris()
```
##### Mutual information
```
# Mutual information between two variables
mi = mi_frame(df)(['F1'], ['F2'])
print(mi)
# Mutual information between two groups of variables (vectors)
mi = mi_frame(df)(['F1','F2'], ['F3','F4', 'F5'])
print(mi)
```
##### Traditional feature selection
```
# Select the two best features according to MRMR (forward), using shorthand
summary, __, __ = MRMR(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2)
print(summary)
# Select the best two features according to JMIM (backward), using default func
__, sel, disc = backward_eliminator(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2, loss=jmim, mi_fun=mi_frame(df))
print(sel)
print(disc)
```
##### Vectorial feature selection
```
# Select the best three features by testing all feat combinations
sel, score = exhaustive_searcher(df, ['F1', 'F2', 'F3', 'F4'], ['F5'], k=2, mi_fun=mi_frame(df))
print(sel)
# Select the best feature vector between two candidates
mifun = mi_frame(df)
aa = ['F1', 'F2']
bb = ['F3', 'F4']
best = aa if mifun(aa, ['F5']) > mifun(bb, ['F5']) else bb
```
### Bibliography
-
Battiti, Roberto. 1994. “Using Mutual Information for Selecting Features in Supervised Neural Net Learning.” IEEE Transactions on Neural Networks 5 (4): 537–50.
-
Yang, H, and John Moody. 1999. “Feature Selection Based on Joint Mutual Information.” In Proceedings of International ICSC Symposium on Advances in Intelligent Data Analysis, 1999:22–25. Citeseer.
-
Fleuret, François. 2004. “Fast Binary Feature Selection with Conditional Mutual Information.” Journal of Machine Learning Research 5 (9).
-
Peng, Hanchuan, Fuhui Long, and Chris Ding. 2005. “Feature Selection Based on Mutual Information Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy.” IEEE Transactions on Pattern Analysis and Machine Intelligence 27 (8): 1226–38.
-
Meyer, Patrick E, and Gianluca Bontempi. 2006. “On the Use of Variable Complementarity for Feature Selection in Cancer Classification.” In Workshops on Applications of Evolutionary Computation, 91–102. Springer.
-
Bennasar, Mohamed, Yulia Hicks, and Rossitza Setchi. 2015. “Feature Selection Using Joint Mutual Information Maximisation.” Expert Systems with Applications 42 (22): 8520–32.
-
Bommert, Andrea, Xudong Sun, Bernd Bischl, Jörg Rahnenführer, and Michel Lang. 2020. “Benchmark for Filter Methods for Feature Selection in High-Dimensional Classification Data.” Computational Statistics & Data Analysis 143: 106839.
-
Kursa, Miron B. 2021. “Praznik: High Performance Information-Based Feature Selection.” SoftwareX 16: 100819.