{"id":13856668,"url":"https://github.com/AnotherSamWilson/miceforest","last_synced_at":"2025-07-13T19:32:20.720Z","repository":{"id":37576975,"uuid":"289387436","full_name":"AnotherSamWilson/miceforest","owner":"AnotherSamWilson","description":"Multiple Imputation with LightGBM in Python","archived":false,"fork":false,"pushed_at":"2024-08-02T00:21:20.000Z","size":5997,"stargazers_count":353,"open_issues_count":8,"forks_count":31,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-11-09T16:46:49.960Z","etag":null,"topics":["data-science","imputed-values","mice-algorithm","python","random-forest"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AnotherSamWilson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-22T00:00:22.000Z","updated_at":"2024-11-08T01:39:48.000Z","dependencies_parsed_at":"2024-04-27T17:31:06.846Z","dependency_job_id":"14b65631-bd55-4a46-8516-98f90039a52c","html_url":"https://github.com/AnotherSamWilson/miceforest","commit_stats":{"total_commits":240,"total_committers":5,"mean_commits":48.0,"dds":"0.033333333333333326","last_synced_commit":"d9359a89204e3b5f10cc02e7e621a22c213e5453"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnotherSamWilson%2Fmiceforest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnotherSamWilson%2Fmiceforest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnotherSamWilson%2Fmiceforest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AnotherSamWilson%2Fmiceforest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AnotherSamWilson","download_url":"https://codeload.github.com/AnotherSamWilson/miceforest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225078667,"owners_count":17417461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","imputed-values","mice-algorithm","python","random-forest"],"created_at":"2024-08-05T03:01:08.133Z","updated_at":"2024-11-22T14:30:35.233Z","avatar_url":"https://github.com/AnotherSamWilson.png","language":"Jupyter Notebook","funding_links":[],"categories":["Python","梯度提升和树模型"],"sub_categories":[],"readme":"[![DOI](https://zenodo.org/badge/289387436.svg)](https://zenodo.org/badge/latestdoi/289387436)\n[![Downloads](https://static.pepy.tech/badge/miceforest)](https://pepy.tech/project/miceforest)\n[![Pypi](https://img.shields.io/pypi/v/miceforest.svg)](https://pypi.python.org/pypi/miceforest)\n[![Conda\nVersion](https://img.shields.io/conda/vn/conda-forge/miceforest.svg)](https://anaconda.org/conda-forge/miceforest)\n[![PyVersions](https://img.shields.io/pypi/pyversions/miceforest.svg?logo=python\u0026logoColor=white)](https://pypi.org/project/miceforest/)  \n[![tests +\nmypy](https://github.com/AnotherSamWilson/miceforest/actions/workflows/run_tests.yml/badge.svg)](https://github.com/AnotherSamWilson/miceforest/actions/workflows/run_tests.yml)\n[![Documentation\nStatus](https://readthedocs.org/projects/miceforest/badge/?version=latest)](https://miceforest.readthedocs.io/en/latest/?badge=latest)\n[![CodeCov](https://codecov.io/gh/AnotherSamWilson/miceforest/branch/master/graphs/badge.svg?branch=master\u0026service=github)](https://codecov.io/gh/AnotherSamWilson/miceforest)\n\u003c!-- [![MIT license](http://img.shields.io/badge/license-MIT-brightgreen.svg)](http://opensource.org/licenses/MIT) --\u003e\n\u003c!-- [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)   --\u003e\n\u003c!-- [![DEV_Version_Badge](https://img.shields.io/badge/Dev-5.6.3-blue.svg)](https://pypi.org/project/miceforest/) --\u003e\n\n\n# miceforest: Fast, Memory Efficient Imputation with LightGBM\n\n\u003ca href='https://github.com/AnotherSamWilson/miceforest'\u003e\u003cimg src='https://i.imgur.com/jJPThrd.png' align=\"right\" height=\"300\" /\u003e\u003c/a\u003e\n\nFast, memory efficient Multiple Imputation by Chained Equations (MICE)\nwith lightgbm. The R version of this package may be found\n[here](https://github.com/FarrellDay/miceRanger).\n\n`miceforest` was designed to be:\n\n  - **Fast**\n      - Uses lightgbm as a backend\n      - Has efficient mean matching solutions.\n      - Can utilize GPU training\n  - **Flexible**\n      - Can impute pandas dataframes\n      - Handles categorical data automatically\n      - Fits into a sklearn pipeline\n      - User can customize every aspect of the imputation process\n  - **Production Ready**\n      - Can impute new, unseen datasets quickly\n      - Kernels are efficiently compressed during saving and loading\n      - Data can be imputed in place to save memory\n      - Can build models on non-missing data\n\nThis document contains a thorough walkthrough of the package,\nbenchmarks, and an introduction to multiple imputation. More information\non MICE can be found in Stef van Buuren’s excellent online book, which\nyou can find\n[here](https://stefvanbuuren.name/fimd/ch-introduction.html).\n\n#### Table of Contents:\n\n  - [Classes](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#classes)\n  - [Basic Usage](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#basic-usage)\n      - [Example](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#basic-usage)\n      - [Customizing LightGBM Parameters](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#customizing-lightgbm-parameters)\n      - [Available Mean Match Schemes](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#adjusting-the-mean-matching-scheme)\n      - [Imputing New Data with Existing Models](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#imputing-new-data-with-existing-models)\n      - [Saving and Loading Kernels](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#saving-and-loading-kernels)\n      - [Implementing sklearn Pipelines](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#saving-and-loading-kernels)\n  - [Advanced Features](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#advanced-features)\n      - [Building Models on Nonmissing Data](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#building-models-on-nonmissing-data)\n      - [Tuning Parameters](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#tuning-parameters)\n      - [On Reproducibility](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#on-reproducibility)\n      - [How to Make the Process Faster](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#how-to-make-the-process-faster)\n      - [Imputing Data In Place](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#imputing-data-in-place)\n  - [Diagnostic Plotting](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#diagnostic-plotting)\n      - [Feature Importance](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#feature-importance)\n      - [Imputed Distributions](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#plot-imputed-distributions)\n  - [Using the Imputed Data](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#using-the-imputed-data)\n  - [The MICE Algorithm](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#the-mice-algorithm)\n      - [Introduction](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#the-mice-algorithm)\n      - [Common Use Cases](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#common-use-cases)\n      - [Predictive Mean Matching](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#predictive-mean-matching)\n      - [Effects of Mean Matching](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#effects-of-mean-matching)\n\n## Installation\n\nThis package can be installed using either pip or conda, through\nconda-forge:\n\n``` bash\n# Using pip\n$ pip install miceforest --no-cache-dir\n\n# Using conda\n$ conda install -c conda-forge miceforest\n```\n\nYou can also download the latest development version from this\nrepository. If you want to install from github with conda, you must\nfirst run `conda install pip git`.\n\n``` bash\n$ pip install git+https://github.com/AnotherSamWilson/miceforest.git\n```\n\n## Classes\n\nmiceforest has 2 main classes which the user will interact with:\n\n  - [`ImputationKernel`](https://miceforest.readthedocs.io/en/latest/ImputationKernel.html)\n    - This class contains the raw data off of which the `mice` algorithm\n    is performed. During this process, models will be trained, and the\n    imputed (predicted) values will be stored. These values can be used\n    to fill in the missing values of the raw data. The raw data can be\n    copied, or referenced directly. Models can be saved, and used to\n    impute new datasets.\n  - [`ImputedData`](https://miceforest.readthedocs.io/en/latest/ImputedData.html)\n    - The result of `ImputationKernel.impute_new_data(new_data)`. This\n    contains the raw data in `new_data` as well as the imputed values.  \n\n\n## Basic Usage\n\nWe will be looking at a few simple examples of imputation. We need to\nload the packages, and define the data:\n\n\n```python\nimport miceforest as mf\nfrom sklearn.datasets import load_iris\nimport pandas as pd\nimport numpy as np\n\n# Load data and introduce missing values\niris = pd.concat(load_iris(as_frame=True,return_X_y=True),axis=1)\niris.rename({\"target\": \"species\"}, inplace=True, axis=1)\niris['species'] = iris['species'].astype('category')\niris_amp = mf.ampute_data(iris,perc=0.25,random_state=1991)\n```\n\nIf you only want to create a single imputed dataset, you can use\n[`ImputationKernel`](https://miceforest.readthedocs.io/en/latest/ImputationKernel.html)\nwith some default settings:\n\n\n```python\n# Create kernel. \nkds = mf.ImputationKernel(\n  iris_amp,\n  random_state=1991\n)\n\n# Run the MICE algorithm for 2 iterations\nkds.mice(2)\n\n# Return the completed dataset.\niris_complete = kds.complete_data()\n```\n\nThere are also an array of plotting functions available, these are\ndiscussed below in the section [Diagnostic\nPlotting](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#diagnostic-plotting).  \n\nWe usually don’t want to impute just a single dataset. In statistics,\nmultiple imputation is a process by which the uncertainty/other effects\ncaused by missing values can be examined by creating multiple different\nimputed datasets.\n[`ImputationKernel`](https://miceforest.readthedocs.io/en/latest/ImputationKernel.html)\ncan contain an arbitrary number of different datasets, all of which have\ngone through mutually exclusive imputation processes:\n\n\n```python\n# Create kernel. \nkernel = mf.ImputationKernel(\n  iris_amp,\n  num_datasets=4,\n  random_state=1\n)\n\n# Run the MICE algorithm for 2 iterations on each of the datasets\nkernel.mice(2)\n\n# Printing the kernel will show you some high level information.\nprint(kernel)\n```\n\n    \n                  Class: ImputationKernel\n                Datasets: 4\n              Iterations: 2\n            Data Samples: 150\n            Data Columns: 5\n       Imputed Variables: 5\n       Modeled Variables: 5\n    All Iterations Saved: True\n            \n\n\nAfter we have run mice, we can obtain our completed dataset directly\nfrom the kernel:\n\n\n```python\ncompleted_dataset = kernel.complete_data(dataset=2)\nprint(completed_dataset.isnull().sum(0))\n```\n\n    sepal length (cm)    0\n    sepal width (cm)     0\n    petal length (cm)    0\n    petal width (cm)     0\n    species              0\n    dtype: int64\n\n\n## Customizing LightGBM Parameters\n\nParameters can be passed directly to lightgbm in several different ways.\nParameters you wish to apply globally to every model can simply be\npassed as kwargs to `mice`:\n\n\n```python\n# Run the MICE algorithm for 1 more iteration on the kernel with new parameters\nkernel.mice(iterations=1, n_estimators=50)\n```\n\nYou can also pass pass variable-specific arguments to\n`variable_parameters` in mice. For instance, let’s say you noticed the\nimputation of the `[species]` column was taking a little longer, because\nit is multiclass. You could decrease the n\\_estimators specifically for\nthat column with:\n\n\n```python\n# Run the MICE algorithm for 2 more iterations on the kernel \nkernel.mice(\n  iterations=1,\n  variable_parameters={'species': {'n_estimators': 25}},\n  n_estimators=50\n)\n\n# Let's get the actual models for these variables:\nspecies_model = kernel.get_model(dataset=0,variable=\"species\")\nsepalwidth_model = kernel.get_model(dataset=0,variable=\"sepal width (cm)\")\n\nprint(\nf\"\"\"Species used {str(species_model.params[\"num_iterations\"])} iterations\nSepal Width used {str(sepalwidth_model.params[\"num_iterations\"])} iterations\n\"\"\"\n)\n```\n\n    Species used 25 iterations\n    Sepal Width used 50 iterations\n    \n\n\nIn this scenario, any parameters specified in `variable_parameters`\ntakes presidence over the kwargs.\n\nSince we can pass any parameters we want to LightGBM, we can completely\ncustomize how our models are built. That includes how the data should be\nmodeled. If your data contains count data, or any other data which can\nbe parameterized by lightgbm, you can simply specify that variable to be\nmodeled with the corresponding objective function.\n\nFor example, let’s pretend `sepal width (cm)` is a count field which can\nbe parameterized by a Poisson distribution. Let’s also change our\nboosting method to gradient boosted trees:\n\n\n```python\n# Create kernel. \ncust_kernel = mf.ImputationKernel(\n  iris_amp,\n  num_datasets=1,\n  random_state=1\n)\n\ncust_kernel.mice(\n  iterations=1, \n  variable_parameters={'sepal width (cm)': {'objective': 'poisson'}},\n  boosting = 'gbdt',\n  min_sum_hessian_in_leaf=0.01\n)\n```\n\nOther nice parameters like `monotone_constraints` can also be passed.\nSetting the parameter `device: 'gpu'` will utilize GPU learning, if\nLightGBM is set up to do this on your machine.\n\n## Adjusting The Mean Matching Scheme\n\nNote: It is probably a good idea to read [this\nsection](https://github.com/AnotherSamWilson/miceforest?tab=readme-ov-file#predictive-mean-matching)\nfirst, to get some context on how mean matching works.\n\nThere are 4 imputation strategies employed by `miceforest`:\n- **Fast** Mean Matching: Available only on binary and categorical variables. Chooses a class randomly based on the predicted probabilities output by lightgbm.\n- **Normal** Mean Matching: Employs mean matching as described in the section below.\n- **Shap** Mean Matching: Runs a nearest neighbor search on the shap values of the bachelor predictions in the shap values of the candidate predictions. Finds the `mean_match_candidates` nearest neighbors, and chooses one randomly as the imputation value.\n- Value Imputation: Uses the value output by lightgbm as the imputation value. Skips mean matching entirely. To use, set `mean_match_candidates = 0`.\n\nHere is the code required to use each method:\n\n\n```python\n# Create kernel. \ncust_kernel = mf.ImputationKernel(\n  iris_amp,\n  num_datasets=1,\n  random_state=1,\n  mean_match_strategy={\n      'sepal length (cm)': 'normal',\n      'sepal width (cm)': 'shap',\n      'species': 'fast',\n  },\n  mean_match_candidates={\n      'petal length (cm)': 0,\n      }\n)\n\ncust_kernel.mice(\n  iterations=1, \n)\n```\n\n## Imputing New Data with Existing Models\n\nMultiple Imputation can take a long time. If you wish to impute a\ndataset using the MICE algorithm, but don’t have time to train new\nmodels, it is possible to impute new datasets using a `ImputationKernel`\nobject. The `impute_new_data()` function uses the models collected by\n`ImputationKernel` to perform multiple imputation without updating the\nmodels at each iteration:\n\n\n```python\n# Our 'new data' is just the first 15 rows of iris_amp\nfrom datetime import datetime\n\n# Define our new data as the first 15 rows\nnew_data = iris_amp.iloc[range(15)].reset_index(drop=True)\n\nstart_t = datetime.now()\nnew_data_imputed = cust_kernel.impute_new_data(new_data=new_data)\nprint(f\"New Data imputed in {(datetime.now() - start_t).total_seconds()} seconds\")\n```\n\n    New Data imputed in 0.040396 seconds\n\n\n## Saving and Loading Kernels\n\nSaving `miceforest` kernels is efficient. During the pickling process, the following steps are taken:\n\n1.  Convert working data to parquet bytes.\n2.  Serialize the kernel.\n4.  Save to a file.\n\nYou can save and load the kernel like any other object using `pickle` or `dill`:\n\n\n\n```python\nfrom tempfile import mkstemp\nimport dill\nnew_file, filename = mkstemp()\n\nwith open(filename, \"wb\") as f:\n    dill.dump(kernel, f)\n\nwith open(filename, \"rb\") as f:\n    kernel_from_pickle = dill.load(f)\n```\n\n## Implementing sklearn Pipelines\n\n`miceforest` kernels can be fit into sklearn pipelines to impute training and scoring\ndatasets:\n\n\n```python\nimport numpy as np\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import Pipeline\nimport miceforest as mf\n\nkernel = mf.ImputationKernel(iris_amp, num_datasets=1, random_state=1)\n\npipe = Pipeline([\n    ('impute', kernel),\n    ('scaler', StandardScaler()),\n])\n\n# The pipeline can be used as any other estimator\n# and avoids leaking the test set into the train set\nX_train_t = pipe.fit_transform(\n    X=iris_amp,\n    y=None,\n    impute__iterations=2\n)\nX_test_t = pipe.transform(new_data)\n\n# Show that neither now have missing values.\nassert not np.any(np.isnan(X_train_t))\nassert not np.any(np.isnan(X_test_t))\n```\n\n# Advanced Features\n\n## Building Models on Nonmissing Data\n\nThe MICE process itself is used to impute missing data in a dataset.\nHowever, sometimes a variable can be fully recognized in the training\ndata, but needs to be imputed later on in a different dataset. It is\npossible to train models to impute variables even if they have no\nmissing values by specifying them in the `variable_schema` parameter. \nIn this case, `variable_schema` is treated as the list of variables \nto train models on.\n\n\n```python\n# Set petal length (cm) in our amputed data \n# to original values with no missing data.\niris_amp['sepal width (cm)'] = iris['sepal width (cm)'].copy()\niris_amp.isnull().sum()\n```\n\n\n\n\n    sepal length (cm)    37\n    sepal width (cm)      0\n    petal length (cm)    37\n    petal width (cm)     37\n    species              37\n    dtype: int64\n\n\n\n\n```python\nkernel = mf.ImputationKernel(\n    data=iris_amp, \n    variable_schema=iris_amp.columns.to_list(), \n    num_datasets=1,\n    random_state=1,\n)\nkernel.mice(1)\n```\n\n\n```python\n# Remember, the dataset we are imputing does have \n# missing values in the sepal width (cm) column\nnew_data.isnull().sum()\n```\n\n\n\n\n    sepal length (cm)    4\n    sepal width (cm)     3\n    petal length (cm)    1\n    petal width (cm)     3\n    species              3\n    dtype: int64\n\n\n\n\n```python\nnew_data_imp = kernel.impute_new_data(new_data)\nnew_data_imp = new_data_imp.complete_data()\n\n# All columns have been imputed.\nnew_data_imp.isnull().sum()\n```\n\n\n\n\n    sepal length (cm)    0\n    sepal width (cm)     0\n    petal length (cm)    0\n    petal width (cm)     0\n    species              0\n    dtype: int64\n\n\n\n## Tuning Parameters\n\n`miceforest` allows you to tune the parameters on a kernel dataset.\nThese parameters can then be used to build the models in future\niterations of mice. In its most simple invocation, you can just call the\nfunction with the desired optimization steps:\n\n\n```python\noptimal_params = kernel.tune_parameters(\n    dataset=0, \n    use_gbdt=True,\n    num_iterations=500,\n    random_state=1,\n)\nkernel.mice(1, variable_parameters=optimal_params)\npd.DataFrame(optimal_params)\n```\n\n\n\n\n\u003cdiv\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003esepal length (cm)\u003c/th\u003e\n      \u003cth\u003epetal length (cm)\u003c/th\u003e\n      \u003cth\u003epetal width (cm)\u003c/th\u003e\n      \u003cth\u003especies\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003eboosting\u003c/th\u003e\n      \u003ctd\u003egbdt\u003c/td\u003e\n      \u003ctd\u003egbdt\u003c/td\u003e\n      \u003ctd\u003egbdt\u003c/td\u003e\n      \u003ctd\u003egbdt\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003edata_sample_strategy\u003c/th\u003e\n      \u003ctd\u003ebagging\u003c/td\u003e\n      \u003ctd\u003ebagging\u003c/td\u003e\n      \u003ctd\u003ebagging\u003c/td\u003e\n      \u003ctd\u003ebagging\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003enum_iterations\u003c/th\u003e\n      \u003ctd\u003e142\u003c/td\u003e\n      \u003ctd\u003e248\u003c/td\u003e\n      \u003ctd\u003e262\u003c/td\u003e\n      \u003ctd\u003e172\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emax_depth\u003c/th\u003e\n      \u003ctd\u003e4\u003c/td\u003e\n      \u003ctd\u003e4\u003c/td\u003e\n      \u003ctd\u003e5\u003c/td\u003e\n      \u003ctd\u003e5\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003enum_leaves\u003c/th\u003e\n      \u003ctd\u003e12\u003c/td\u003e\n      \u003ctd\u003e17\u003c/td\u003e\n      \u003ctd\u003e2\u003c/td\u003e\n      \u003ctd\u003e19\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emin_data_in_leaf\u003c/th\u003e\n      \u003ctd\u003e2\u003c/td\u003e\n      \u003ctd\u003e2\u003c/td\u003e\n      \u003ctd\u003e15\u003c/td\u003e\n      \u003ctd\u003e5\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emin_sum_hessian_in_leaf\u003c/th\u003e\n      \u003ctd\u003e0.1\u003c/td\u003e\n      \u003ctd\u003e0.1\u003c/td\u003e\n      \u003ctd\u003e0.1\u003c/td\u003e\n      \u003ctd\u003e0.1\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003emin_gain_to_split\u003c/th\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003ebagging_fraction\u003c/th\u003e\n      \u003ctd\u003e0.580973\u003c/td\u003e\n      \u003ctd\u003e0.501521\u003c/td\u003e\n      \u003ctd\u003e0.586709\u003c/td\u003e\n      \u003ctd\u003e0.795465\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003efeature_fraction_bynode\u003c/th\u003e\n      \u003ctd\u003e0.922566\u003c/td\u003e\n      \u003ctd\u003e0.299912\u003c/td\u003e\n      \u003ctd\u003e0.503182\u003c/td\u003e\n      \u003ctd\u003e0.237637\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003ebagging_freq\u003c/th\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003everbosity\u003c/th\u003e\n      \u003ctd\u003e-1\u003c/td\u003e\n      \u003ctd\u003e-1\u003c/td\u003e\n      \u003ctd\u003e-1\u003c/td\u003e\n      \u003ctd\u003e-1\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003elearning_rate\u003c/th\u003e\n      \u003ctd\u003e0.02\u003c/td\u003e\n      \u003ctd\u003e0.02\u003c/td\u003e\n      \u003ctd\u003e0.02\u003c/td\u003e\n      \u003ctd\u003e0.02\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eobjective\u003c/th\u003e\n      \u003ctd\u003eregression\u003c/td\u003e\n      \u003ctd\u003eregression\u003c/td\u003e\n      \u003ctd\u003eregression\u003c/td\u003e\n      \u003ctd\u003emulticlass\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003enum_class\u003c/th\u003e\n      \u003ctd\u003eNaN\u003c/td\u003e\n      \u003ctd\u003eNaN\u003c/td\u003e\n      \u003ctd\u003eNaN\u003c/td\u003e\n      \u003ctd\u003e3\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\nThis will perform 10 fold cross validation on random samples of\nparameters. By default, all variables models are tuned.  \n\nThe parameter tuning is pretty flexible. If you wish to set some model\nparameters static, or to change the bounds that are searched in, you can\nsimply pass this information to either the `variable_parameters`\nparameter, `**kwbounds`, or both:\n\n\n```python\noptimal_params = kernel.tune_parameters(\n  dataset=0,\n  variables = ['sepal width (cm)','species','petal width (cm)'],\n  variable_parameters = {\n    'sepal width (cm)': {'bagging_fraction': 0.5},\n    'species': {'bagging_freq': (5,10)}\n  },\n  use_gbdt=True,\n  optimization_steps=5,\n  extra_trees = [True, False]\n)\n\nkernel.mice(1, variable_parameters=optimal_params)\n```\n\nIn this example, we did a few things - we specified that only `sepal\nwidth (cm)`, `species`, and `petal width (cm)` should be tuned. We also\nspecified some specific parameters in `variable_parameters`. Notice that\n`bagging_fraction` was passed as a scalar, `0.5`. This means that, for\nthe variable `sepal width (cm)`, the parameter `bagging_fraction` will\nbe set as that number and not be tuned. We did the opposite for\n`bagging_freq`. We specified bounds that the process should search in.\nWe also passed the argument `extra_trees` as a list. Since it was passed\nto \\*\\*kwbounds, this parameter will apply to all variables that are\nbeing tuned. Passing values as a list tells the process that it should\nrandomly sample values from the list, instead of treating them as set of\ncounts to search within.\n\nAdditionally, we set `use_gbdt=True`. This switches the process to use\ngradient boosted trees, instead of random forests. Typically, gradient\nboosted trees will perform better. The optimal `num_iterations` is also\ndetermined by early stopping in cross validation.\n\nThe tuning process follows these rules for different parameter values it\nfinds:\n\n  - Scalar: That value is used, and not tuned.  \n  - Tuple: Should be length 2. Treated as the lower and upper bound to\n    search in.  \n  - List: Treated as a distinct list of values to try randomly.\n\n\n## On Reproducibility\n\n`miceforest` allows for different “levels” of reproducibility, global\nand record-level.\n\n##### **Global Reproducibility**\n\nGlobal reproducibility ensures that the same values will be imputed if\nthe same code is run multiple times. To ensure global reproducibility,\nall the user needs to do is set a `random_state` when the kernel is\ninitialized.\n\n##### **Record-Level Reproducibility**\n\nSometimes we want to obtain reproducible imputations at the record\nlevel, without having to pass the same dataset. This is possible by\npassing a list of record-specific seeds to the `random_seed_array`\nparameter. This is useful if imputing new data multiple times, and you\nwould like imputations for each row to match each time it is imputed.\n\n\n\n```python\n# Define seeds for the data, and impute iris\nimport numpy as np\nrandom_seed_array = np.random.randint(0, 9999, size=iris_amp.shape[0], dtype='uint32')\niris_imputed = kernel.impute_new_data(\n    iris_amp,\n    random_state=4,\n    random_seed_array=random_seed_array\n)\n\n# Select a random sample\nnew_inds = np.random.choice(150, size=15)\nnew_data = iris_amp.loc[new_inds].reset_index(drop=True)\nnew_seeds = random_seed_array[new_inds]\nnew_imputed = kernel.impute_new_data(\n    new_data,\n    random_state=4,\n    random_seed_array=new_seeds\n)\n\n# We imputed the same values for the 15 values each time,\n# because each record was associated with the same seed.\nassert new_imputed.complete_data(0).equals(\n    iris_imputed.complete_data(0).loc[new_inds].reset_index(drop=True)\n)\n```\n\n## How to Make the Process Faster\n\nMultiple Imputation is one of the most robust ways to handle missing\ndata - but it can take a long time. There are several strategies you can\nuse to decrease the time a process takes to run:\n\n  - Decrease `data_subset`. By default all non-missing datapoints for\n    each variable are used to train the model and perform mean matching.\n    This can cause the model training nearest-neighbors search to take a\n    long time for large data. A subset of these points can be searched\n    instead by using `data_subset`.  \n  - If categorical columns are taking a long time, you can set \n    `mean_match_strategy=\"fast\"`. You can also set different parameters\n    specifically for categorical columns, like smaller `bagging_fraction` \n    or `num_iterations`, or try grouping the categories before they are\n    imputed. Model training time for categorical variables is linear with\n    the number of distinct categories. \n  - Decrease `mean_match_candidates`. The maximum number of neighbors\n    that are considered with the default parameters is 10. However, for\n    large datasets, this can still be an expensive operation. Consider\n    explicitly setting `mean_match_candidates` lower. Setting \n    `mean_match_candidates=0` will skip mean matching entirely, and \n    just use the lightgbm predictions as the imputation values.\n  - Use different lightgbm parameters. lightgbm is usually not the\n    problem, however if a certain variable has a large number of\n    classes, then the max number of trees actually grown is (\\# classes)\n    \\* (n\\_estimators). You can specifically decrease the bagging\n    fraction or n\\_estimators for large multi-class variables, or grow\n    less trees in general.  \n\n## Imputing Data In Place\n\nIt is possible to run the entire process without copying the dataset. If\n`copy_data=False`, then the data is referenced directly:\n\n\n\n```python\nkernel_inplace = mf.ImputationKernel(\n  iris_amp,\n  num_datasets=1,\n  copy_data=False,\n  random_state=1,\n)\nkernel_inplace.mice(2)\n```\n\nNote, that this probably won’t (but could) change the original dataset\nin undesirable ways. Throughout the `mice` procedure, imputed values are\nstored directly in the original data. At the end, the missing values are\nput back as `np.NaN`.\n\nWe can also complete our original data in place. This is useful if the dataset is large, and copies can’t be made in\nmemory:\n\n\n```python\nkernel_inplace.complete_data(dataset=0, inplace=True)\nprint(iris_amp.isnull().sum(0))\n```\n\n    sepal length (cm)    0\n    sepal width (cm)     0\n    petal length (cm)    0\n    petal width (cm)     0\n    species              0\n    dtype: int64\n\n\n# Diagnostic Plotting\n\nAs of now, there are 2 diagnostic plot available. More coming soon!\n\n### Feature Importance\n\n\n```python\nkernel.plot_feature_importance(dataset=0)\n```\n\n\n    \n![png](README_files/README_48_0.png)\n    \n\n\n### Plot Imputed Distributions\n\n\n```python\nkernel.plot_imputed_distributions()\n```\n\n\n    \n![png](README_files/README_50_0.png)\n    \n\n\n## Using the Imputed Data\n\nTo return the imputed data simply use the `complete_data` method:\n\n\n```python\ndataset_1 = kernel.complete_data(0)\n```\n\nThis will return a single specified dataset. Multiple datasets are\ntypically created so that some measure of confidence around each\nprediction can be created.\n\nSince we know what the original data looked like, we can cheat and see\nhow well the imputations compare to the original data:\n\n\n```python\nacclist = []\niterations = kernel.iteration_count()+1\nfor iteration in range(iterations):\n    species_na_count = kernel.na_counts['species']\n    compdat = kernel.complete_data(dataset=0,iteration=iteration)\n    \n    # Record the accuract of the imputations of species.\n    acclist.append(\n      round(1-sum(compdat['species'] != iris['species'])/species_na_count,2)\n    )\n\n# acclist shows the accuracy of the imputations over the iterations.\nacclist = pd.Series(acclist).rename(\"Species Imputation Accuracy\")\nacclist.index = range(iterations)\nacclist.index.name = \"Iteration\"\nacclist\n```\n\n\n\n\n    Iteration\n    0    0.35\n    1    0.81\n    2    0.81\n    3    0.78\n    Name: Species Imputation Accuracy, dtype: float64\n\n\n\nIn this instance, we went from a low accuracy (what is expected with\nrandom sampling) to a much higher accuracy.\n\n## The MICE Algorithm\n\nMultiple Imputation by Chained Equations ‘fills in’ (imputes) missing\ndata in a dataset through an iterative series of predictive models. In\neach iteration, each specified variable in the dataset is imputed using\nthe other variables in the dataset. These iterations should be run until\nit appears that convergence has been met.\n\n\u003cimg src=\"https://i.imgur.com/2L403kU.png\" style=\"display: block; margin: auto;\" /\u003e\n\nThis process is continued until all specified variables have been\nimputed. Additional iterations can be run if it appears that the average\nimputed values have not converged, although no more than 5 iterations\nare usually necessary.\n\n### Common Use Cases\n\n##### **Data Leakage:**\n\nMICE is particularly useful if missing values are associated with the\ntarget variable in a way that introduces leakage. For instance, let’s\nsay you wanted to model customer retention at the time of sign up. A\ncertain variable is collected at sign up or 1 month after sign up. The\nabsence of that variable is a data leak, since it tells you that the\ncustomer did not retain for 1 month.\n\n##### **Funnel Analysis:**\n\nInformation is often collected at different stages of a ‘funnel’. MICE\ncan be used to make educated guesses about the characteristics of\nentities at different points in a funnel.\n\n##### **Confidence Intervals:**\n\nMICE can be used to impute missing values, however it is important to\nkeep in mind that these imputed values are a prediction. Creating\nmultiple datasets with different imputed values allows you to do two\ntypes of inference:\n\n  - Imputed Value Distribution: A profile can be built for each imputed\n    value, allowing you to make statements about the likely distribution\n    of that value.  \n  - Model Prediction Distribution: With multiple datasets, you can build\n    multiple models and create a distribution of predictions for each\n    sample. Those samples with imputed values which were not able to be\n    imputed with much confidence would have a larger variance in their\n    predictions.\n\n\n## Predictive Mean Matching\n\n`miceforest` can make use of a procedure called predictive mean matching\n(PMM) to select which values are imputed. PMM involves selecting a\ndatapoint from the original, nonmissing data (candidates) which has a\npredicted value close to the predicted value of the missing sample\n(bachelors). The closest N (`mean_match_candidates` parameter) values\nare selected, from which a value is chosen at random. This can be\nspecified on a column-by-column basis. Going into more detail from our\nexample above, we see how this works in practice:\n\n\u003cimg src=\"https://i.imgur.com/3DBCXnL.png\" style=\"display: block; margin: auto;\" /\u003e\n\nThis method is very useful if you have a variable which needs imputing\nwhich has any of the following characteristics:\n\n  - Multimodal  \n  - Integer  \n  - Skewed\n\n\n### Effects of Mean Matching\n\nAs an example, let’s construct a dataset with some of the above\ncharacteristics:\n\n\n```python\nrandst = np.random.RandomState(1991)\n# random uniform variable\nnrws = 1000\nuniform_vec = randst.uniform(size=nrws)\n\ndef make_bimodal(mean1,mean2,size):\n    bimodal_1 = randst.normal(size=nrws, loc=mean1)\n    bimodal_2 = randst.normal(size=nrws, loc=mean2)\n    bimdvec = []\n    for i in range(size):\n        bimdvec.append(randst.choice([bimodal_1[i], bimodal_2[i]]))\n    return np.array(bimdvec)\n\n# Make 2 Bimodal Variables\nclose_bimodal_vec = make_bimodal(2,-2,nrws)\nfar_bimodal_vec = make_bimodal(3,-3,nrws)\n\n\n# Highly skewed variable correlated with Uniform_Variable\nskewed_vec = np.exp(uniform_vec*randst.uniform(size=nrws)*3) + randst.uniform(size=nrws)*3\n\n# Integer variable correlated with Close_Bimodal_Variable and Uniform_Variable\ninteger_vec = np.round(uniform_vec + close_bimodal_vec/3 + randst.uniform(size=nrws)*2)\n\n# Make a DataFrame\ndat = pd.DataFrame(\n    {\n    'uniform_var':uniform_vec,\n    'close_bimodal_var':close_bimodal_vec,\n    'far_bimodal_var':far_bimodal_vec,\n    'skewed_var':skewed_vec,\n    'integer_var':integer_vec\n    }\n)\n\n# Ampute the data.\nampdat = mf.ampute_data(dat,perc=0.25,random_state=randst)\n```\n\n\n```python\nimport plotnine as p9\nimport itertools\n\ndef plot_matrix(df, columns):\n    pdf = []\n    for a1, b1 in itertools.combinations(columns, 2):\n        for (a,b) in ((a1, b1), (b1, a1)):\n            sub = df[[a, b]].rename(columns={a: \"x\", b: \"y\"}).assign(a=a, b=b)\n            pdf.append(sub)\n\n    g = (\n        p9.ggplot(pd.concat(pdf))\n            + p9.geom_point(p9.aes('x','y'))\n            + p9.facet_grid('b~a', scales='free')\n            + p9.theme(figure_size=(7, 7))\n            + p9.xlab(\"\") + p9.ylab(\"\")\n    )\n    return g\n\nplot_matrix(dat, dat.columns)\n```\n\n\n    \n![png](README_files/README_60_0.png)\n    \n\n\nWe can see how our variables are distributed and correlated in the graph\nabove. Now let’s run our imputation process twice, once using mean\nmatching, and once using the model prediction.\n\n\n```python\nkernel_mean_match = mf.ImputationKernel(\n    data=ampdat,\n    num_datasets=3,\n    mean_match_candidates=5,\n    random_state=1\n)\nkernel_mean_match.mice(2)\nkernel_no_mean_match = mf.ImputationKernel(\n    data=ampdat,\n    num_datasets=3,\n    mean_match_candidates=0,\n    random_state=1\n)\nkernel_no_mean_match.mice(2)\n```\n\n\n```python\nkernel_mean_match.plot_imputed_distributions()\n```\n\n\n    \n![png](README_files/README_63_0.png)\n    \n\n\n\n```python\nkernel_no_mean_match.plot_imputed_distributions()\n```\n\n\n    \n![png](README_files/README_64_0.png)\n    \n\n\nYou can see the effects that mean matching has, depending on the\ndistribution of the data. Simply returning the value from the model\nprediction, while it may provide a better ‘fit’, will not provide\nimputations with a similair distribution to the original. This may be\nbeneficial, depending on your goal.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAnotherSamWilson%2Fmiceforest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAnotherSamWilson%2Fmiceforest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAnotherSamWilson%2Fmiceforest/lists"}