https://github.com/abhayspawar/featexp

Feature exploration for supervised learning
https://github.com/abhayspawar/featexp

data-exploration data-science feature-engineering machine-learning visualization

Last synced: 2 months ago
JSON representation

Feature exploration for supervised learning

• Host: GitHub
• URL: https://github.com/abhayspawar/featexp
• Owner: abhayspawar
• License: mit
• Created: 2018-09-16T19:29:55.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2021-02-04T03:15:41.000Z (almost 4 years ago)
• Last Synced: 2024-10-29T20:46:54.403Z (3 months ago)
• Topics: data-exploration, data-science, feature-engineering, machine-learning, visualization
• Language: Jupyter Notebook
• Size: 4.95 MB
• Stars: 763
• Watchers: 22
• Forks: 163
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# featexp

Feature exploration for supervised learning. Helps with feature understanding, identifying noisy features, feature debugging, leakage detection and model monitoring.

## Installation

```pip install featexp```

## Using featexp

Detailed [Medium post](https://towardsdatascience.com/my-secret-sauce-to-be-in-top-2-of-a-kaggle-competition-57cff0677d3c) on using featexp. Translations from web: [Chinese](https://www.jiqizhixin.com/articles/2018-11-06-7), [Russian](https://neurohive.io/ru/tutorial/kak-popast-v-top-2-sorevnovanija-kaggle/)

featexp draws plots similar to partial dependence plots, but directly from data instead of using a trained model like current implementations of pdp do. Since it draws plots from data directly, it helps with understanding the features well and building better ML models.

```bash

from featexp import get_univariate_plots

get_univariate_plots(data=data_train, target_col='target', data_test=data_test, features_list=['DAYS_EMPLOYED'])

# data_test and features_list are optional.

# Draws plots for all columns if features_list not passed

# Draws only train data plots if no test_data passed

```

![Output1](demo/sample_outputs/days_employed.png)

featexp bins a feature into equal population bins and shows the mean value of the dependent variable (target) in each bin. Here's how to read these plots:

  1. The trend plot on the left helps you understand the relationship between target and feature.

  2. Population distribution helps you make sure the feature is correct.

  3. Also, shows the number of trend direction changes and the correlation between train and test trend which can be used to identify noisy features. A high number of trend changes or low trend correlation implies high noise.

Example of a noisy feature: Has low trend correlation

![Noisy feature](demo/sample_outputs/noisy_feature.png)

### Getting binned feature stats

Returns mean target and population in each bin of a feature

```bash

from featexp import univariate_plotter

binned_data_train, binned_data_test = univariate_plotter(data=data_train, target_col='target', feature='DAYS_EMPLOYED', data_test=data_test)

# For only train data

binned_data_train = univariate_plotter(data=data_train, target_col='target', feature='DAYS_EMPLOYED')

```

### Getting stats for all features

Returns trend changes and trend correlation for all features in a dataframe

```bash

from featexp import get_trend_stats_feature

stats = get_trend_stats(data=data_train, target_col='target', data_test=data_test)

# data_test is optional. If not passed, trend correlations aren't calculated.

```

Returns a dataframe with trend changes and trend correlation which can be used for dropping the noisy features, etc.

![Output1](demo/sample_outputs/stats_output.png)

### Leakage detection

It helps with identifying why a feature is leaky which helps with debugging.

![Leaky feature](demo/sample_outputs/leaky_feature.png)

Nulls have 0% mean target and 100% mean target in other bins. Implies this feature is populated only for target = 1.

### Citing featexp

![s](https://zenodo.org/badge/doi/10.5281/zenodo.2616757.svg)