{"id":18561609,"url":"https://github.com/ajayarunachalam/meta-self-learner","last_synced_at":"2025-05-15T16:34:50.655Z","repository":{"id":57441109,"uuid":"361534133","full_name":"ajayarunachalam/meta-self-learner","owner":"ajayarunachalam","description":"Meta Ensemble Self-Learning Model with Optimization","archived":false,"fork":false,"pushed_at":"2022-03-18T02:53:05.000Z","size":1034,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-22T20:19:53.422Z","etag":null,"topics":["ensemble-learning","generalization-error","metalearning","model-pipeline","optimization","optimization-methods","predictive-modeling","self-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ajayarunachalam.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-25T20:43:14.000Z","updated_at":"2022-03-18T02:53:08.000Z","dependencies_parsed_at":"2022-09-02T06:42:59.000Z","dependency_job_id":null,"html_url":"https://github.com/ajayarunachalam/meta-self-learner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajayarunachalam%2Fmeta-self-learner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajayarunachalam%2Fmeta-self-learner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajayarunachalam%2Fmeta-self-learner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajayarunachalam%2Fmeta-self-learner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ajayarunachalam","download_url":"https://codeload.github.com/ajayarunachalam/meta-self-learner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254377729,"owners_count":22061212,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ensemble-learning","generalization-error","metalearning","model-pipeline","optimization","optimization-methods","predictive-modeling","self-learning"],"created_at":"2024-11-06T22:07:23.880Z","updated_at":"2025-05-15T16:34:50.635Z","avatar_url":"https://github.com/ajayarunachalam.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"meta-self-learner\n=================\n\n**Meta-Self-Learner**\n\npypi: https://pypi.org/project/meta-self-learner/\n\nAbout\n=====\n\nMeta-Self-Learn aims to provide several ensemble learners functionality for quick predictive modeling. Generally, predictions becomes unreliable when the input sample is out of the training distribution, bias to data distribution or error prone to noise, and so on. Current approaches require changes to the network architecture, fine tuning, balanced data, increasing model size, etc. Also, the selection of the algorithm plays a vital role, while the scalability and learning ability decrease with the complex datasets. With this package, I develop an ensemble framework for minimizing generalization error in the learning algorithm irrespective of the data distribution, number of classes, choice of algorithm, number of models, complexity of the datasets. So, in summary we can be able to generalize better. One of the key take-away of this package is building models in more stable fashion while minimizing under-fitting/over-fitting which is very critical to the overall outcome. \n\nThe first layer comprises of several individual classifiers. We have used the base classifiers namely Logistic Regression (LR), K-Nearest Neighbor (KNN), Random Forest (RF), Support Vector Machine (SVM), Extra Tree Classifier (ETC), and Gradient Boosting Machines (GBM). The two self-learners (i.e., Ensemble-1 and Ensemble-2) aim to find the optimal coefficients that minimize the objective function, i.e., the log-loss function. With given set of predictions obtained in the previous layer, the two meta-learner define two different linear problems, and aim to optimize the objective function to find the optimal coefficients that lower's the loss.\n\nBasic workflow\n--------------\n\nThe designed model pipeline workflow is as given in the figure. \n\n.. image:: figures/images/MODELING_PIPELINE.png\n\n\nThe pre-processed data is input to Layer-1 of the model. 'T' and 'P' represent training data and predictions, respectively. In Layer-1, many standalone base learns are used. Input to Layer-2 includes the predictions from previous Layer-1. Two meta-self-learner ensemble schemes are used. Layer-3 combines the results from the Layer-2 predictions as a simple weighted average (WA). Model evaluation, and result interpretation is done finally in the last stage of the pipeline.\n\nThe details of the meta-self learning architecture is as follows:-\n\nLAYER-1:\n--------\nSix classifiers are used (LR, SVM, RF, ETC, GBM, and KNN).\nAll the classifiers are applied twice:\n1) The classifiers are trained on (X_train, y_train), and used to predict the class probabilities of (X_valid).\n2) The classifiers are trained on (X = (X_train + X_valid), y= (y_train + y_valid)) and used to predict the class probabilities of (X_test).\n\nLAYER-2:\n--------\nThe prediction from the previous layer on X_valid are concatenated, and used to create a new training set (XV, y_valid). The predictions on X_test are concatenated to create new test set (XT, y_test). The two proposed ensemble methods, and their calibrated versions are trained on (XV, y_valid), and used to predict the class probabilities of (XT).\n\nLAYER-3:\n--------\n\nThe predictions from the previous layer-2 are then linearly combined using weighted average.\n\nIn this way, a hybrid model is designed \u0026 deployed, where the predictions of the standalone classifier are combined by meta self learner methods, thereby reducing the risk of under-fitting/over-fitting.\n\n\nRequirements\n============\n\n-  **Python 3.6.x**\n-  NumPy[\u003e=1.9.0]\n-  SciPy[\u003e=0.14.0]\n-  Scikit-learn[\u003e=0.16]\n-  Pandas[\u003e=0.23.0]\n-  Tabulate[\u003e=0.8.9]\n-  Xgboost[\u003e=1.4.1]\n-  Six\n-  Matplotlib\n-  Plot-metric\n\n\nQuickly Setup package with automation scripts\n=============================================\n\n.. code:: bash\n\n    sudo bash setup.sh\n\nInstallation\n------------\nUsing pip:\n\n.. code:: sh\n\n    pip install meta-self-learner\n\nUsing notebook:\n\n.. code:: sh\n\n    !pip install meta-self-learner\n\n\nGet started\n===========\n\nThe META-SELF-LEARNER introduces layered network architecture:\n\n-  **DEMO:**\n\nExample MultiClass Classification\n---------------------------------\n\nSimple meta-self-learner\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nLet's load a simple dataset, make a train \u0026 test set and deploy the Meta Self Learner Model:\n\n.. code:: python\n\n    __author__ = 'Ajay Arunachalam'\n    __version__ = '0.0.1'\n    __date__ = '25.4.2021'\n\n    # load libraries\n    import os\n    import pandas as pd\n    import numpy as np\n    import matplotlib.pylab as plt\n    %matplotlib inline\n    from sklearn.datasets import load_digits # example dataset\n    from sklearn.model_selection import train_test_split\n    from sklearn.linear_model import LogisticRegression\n    from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, ExtraTreesClassifier\n    from sklearn.svm import SVC\n    from sklearn.neighbors import KNeighborsClassifier\n    from sklearn.calibration import CalibratedClassifierCV\n    from sklearn.cross_validation import train_test_split\n    from sklearn.linear_model import LogisticRegressionCV\n    from tabulate import tabulate\n    from xgboost import XGBClassifier\n\n    # load built package functions\n    from msl.MetaLearning import *\n    from plot_metric.functions import MultiClassClassification\n    from msl.cf_matrix import make_confusion_matrix\n    \n    #fixing random state\n    random_state=123\n\n    # Load dataset (we just selected 4 classes of digits)\n    X, Y = load_digits(n_class=4, return_X_y=True)\n\n    print(f'Predictors: {X}')\n\n    print(f'Outcome: {Y}')\n\n    # Add noisy features to make the problem more harder\n    random_state = np.random.RandomState(123)\n    n_samples, n_features = X.shape\n    X = np.c_[X, random_state.randn(n_samples, 1000 * n_features)]\n\n    ## Spliting data into train and test sets.\n    X, X_test, y, y_test = train_test_split(X, Y, test_size=0.2, \n                                            random_state=123)\n        \n    ## Spliting train data into training and validation sets.\n    X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.25, \n                                                          random_state=1)\n\n    print('Data shape:')\n    print('X_train: %s, X_valid: %s, X_test: %s \\n' %(X_train.shape, X_valid.shape, \n                                                      X_test.shape))\n\n    # Create list to store logloss of individual classifiers (single classifier) \u0026 meta self-learners\n    ll_sc, ll_ensemble1, ll_ensemble2, ll_ensemble1_cc, ll_ensemble2_cc, ll_ensemble3, ll_lr, ll_gb = [[] for i in range(8)]\n\n    #Defining the classifiers\n    clfs = {'LR'  : LogisticRegression(random_state=random_state), \n            'SVM' : SVC(probability=True, random_state=random_state), \n            'RF'  : RandomForestClassifier(n_estimators=100, n_jobs=-1, \n                                           random_state=random_state), \n           'GBM' : GradientBoostingClassifier(n_estimators=50, \n                                              random_state=random_state), \n            'ETC' : ExtraTreesClassifier(n_estimators=100, n_jobs=-1, \n                                         random_state=random_state),\n            'KNN' : KNeighborsClassifier(n_neighbors=30)}\n        \n    #predictions on the validation and test sets\n    p_valid = []\n    p_test = []\n\n    ########################### LAYER 1 ##########################\n    '''\n    First layer (individual classifiers)\n    All classifiers are applied twice:\n    Training on (X_train, y_train) and predicting on (X_valid)\n    Training on (X, y) and predicting on (X_test)\n    We can add / remove classifiers or change parameter values to see the effect on final results.\n    '''\n    print('Performance of individual classifiers (1st layer) on X_test')   \n    print('------------------------------------------------------------')\n\n    for lg, clf in clfs.items():\n        #First run. Training on (X_train, y_train) and predicting on X_valid.\n        clf.fit(X_train, y_train.ravel())\n        yv = clf.predict_proba(X_valid)\n        p_valid.append(yv)\n\n        # second run. Training on (X, y) and predicting on X_test.\n        clf.fit(X, y.ravel())\n        yt= clf.predict_proba(X_test)\n        p_test.append(yt)\n\n        # print the performance for each classifier\n        print('{:10s} {:2s} {:1.7f}'. format('%s:' %(lg), 'logloss =\u003e', log_loss(y_test, yt)))\n        #Saving the logloss score\n        ll_sc.append(log_loss(y_test, yt)) #Saving the logloss score\n    print('')\n\n    # Configure the number of class to input into the model\n\n    NUM_CLASS = MetaEnsemble.set_config(NUM_CLASS=4) # Enter your number of classes in the dataset here\n\n    # Using Ensemble1 and Ensemble2 in a THREE-LAYERED META LEARNER architecture.\n\n    ########################### LAYER 2 ##########################\n    '''\n    (optimization based ensembles)\n    Predictions on X_valid are used as training set (XV) and predictions on X_test are used as test set (XT). \n    Ensemble1, Ensemble2 and their calibrated versions are applied.\n    '''\n    print('Performance of optimization based meta self-learners (2nd layer) on X_test')\n    print('------------------------------------------------------------')\n    #Creating the data for the 2nd layer.\n\n    XV = np.hstack(p_valid)\n    XT = np.hstack(p_test)\n\n    # Ensemble1\n\n    en1 = MetaEnsemble.Ensemble_one(NUM_CLASS) # as we have 26 classes n_classes=26\n    en1.fit(XV, y_valid.ravel())\n    w_en1 = en1.w\n    y_en1 = en1.predict_proba(XT)\n    print('{:20s} {:2s} {:1.7f}'.format('Ensemble1:', 'logloss =\u003e', log_loss(y_test, y_en1)))\n    ll_ensemble1.append(log_loss(y_test, y_en1)) #Saving the logloss score\n\n    #Calibrated version of Ensemble1\n\n    cc_en1 = CalibratedClassifierCV(en1, method='isotonic')\n    cc_en1.fit(XV,y_valid.ravel())\n    y_cc_en1 = cc_en1.predict_proba(XT)\n    print('{:20s} {:2s} {:1.7f}'.format('Calibrated_Ensemble1:', 'logloss =\u003e', log_loss(y_test, y_cc_en1)))\n    ll_ensemble1_cc.append(log_loss(y_test, y_cc_en1)) #Saving the logloss score\n\n    # Ensemble2\n\n    en2 = MetaEnsemble.Ensemble_two(NUM_CLASS) # as we have 26 classes n_classes=26\n    en2.fit(XV,y_valid.ravel())\n    w_en2 = en2.w\n    y_en2 = en2.predict_proba(XT)\n    print('{:20s} {:2s} {:1.7f}'.format('Ensemble2:', 'logloss =\u003e', log_loss(y_test, y_en2)))\n    ll_ensemble2.append(log_loss(y_test, y_en2)) #Saving the logloss score\n\n    #Calibrated version of Ensemble2\n\n    cc_en2 = CalibratedClassifierCV(en2, method='isotonic')\n    cc_en2.fit(XV,y_valid.ravel())\n    y_cc_en2 = cc_en2.predict_proba(XT)\n    print('{:20s} {:2s} {:1.7f}'.format('Calibrated_Ensemble2:', 'logloss =\u003e', log_loss(y_test, y_cc_en2)))\n    ll_ensemble2_cc.append(log_loss(y_test, y_cc_en2)) #Saving the logloss score\n    print('')\n\n    ############# Third layer (weighted average) ######################################\n    # Simple weighted average of the previous 4 predictions.\n    print('Performance of agggregation of the self-learners (3rd layer) on X_test')\n    print('------------------------------------------------------------')\n    y_thirdlayer = (y_en1 * 4./9.) + (y_cc_en1 * 2./9.) + (y_en2 * 2./9.) + (y_cc_en2 * 1./9.)\n    print('{:20s} {:2s} {:1.7f}'.format('3rd_layer:', 'logloss =\u003e', log_loss(y_test, y_thirdlayer)))\n    ll_ensemble3.append(log_loss(y_test, y_thirdlayer))\n\n    '''\n    # Plotting the weights of each ensemble\n    In the case of Ensemble1, there is a weight for each prediction \n    and in the case of Ensemble2 there is a weight for each class for each prediction.\n    '''\n    from tabulate import tabulate\n    print(' Weights of Ensemble1:')\n    print('|---------------------------------------------|')\n    wA = np.round(w_en1, decimals=2).reshape(1,-1)\n    print(tabulate(wA, headers=clfs.keys(), tablefmt=\"orgtbl\"))\n    print('')\n    print(' Weights of Ensemble2:')\n    print('|-------------------------------------------------------------------------------------------|')\n    wB = np.round(w_en2.reshape((-1,NUM_CLASS)), decimals=2) # 26 is no. of classes (NUM_CLASS)\n    wB = np.hstack((np.array(list(clfs.keys()), dtype=str).reshape(-1,1), wB))\n    print(tabulate(wB, headers=['y%s'%(i) for i in range(NUM_CLASS)], tablefmt=\"orgtbl\"))\n\n    '''\n    Comparing the ensemble results with sklearn LogisticRegression based stacking of classifiers.\n    Both techniques Ensemble1 and Ensemble2 optimizes an objective function. \n    In this experiment I am using the multi-class logloss as objective function. \n    Therefore, the two proposed methods basically become implementations of LogisticRegression. The following\n    code allows to compare the results of sklearn implementation of LogisticRegression with the proposed ensembles.\n    '''\n    #By default the best C parameter is obtained with a cross-validation approach, doing grid search with\n    #10 values defined in a logarithmic scale between 1e-4 and 1e4.\n    #Change parameters to see how they affect the final results.\n\n    # LogisticRegression\n    lr = LogisticRegressionCV(Cs=10, dual=False, fit_intercept=True,\n        intercept_scaling=1.0, max_iter=100,\n        multi_class='ovr', n_jobs=1, penalty='l2',\n        random_state=random_state,\n        solver='lbfgs', tol=0.0001)\n\n    lr.fit(XV, y_valid.ravel())\n    y_lr = lr.predict_proba(XT)\n    print('{:20s} {:2s} {:1.7f}'.format('Logistic_Regression:', 'logloss =\u003e', log_loss(y_test, y_lr)))\n    ll_lr.append(log_loss(y_test, y_lr)) #Saving the logloss score\n    print('')\n\n    '''\n    Comparing the ensemble results with sklearn GradientBoost based stacking of classifiers.\n    Both techniques Ensemble1 and Ensemble2 optimizes an objective function. \n    In this experiment I am using the multi-class logloss as objective function. \n    '''\n    from xgboost import XGBClassifier\n    # Gradient boosting\n    xgb = XGBClassifier(max_depth=5, learning_rate=0.1,n_estimators=10000, objective='multi:softprob',seed=random_state)\n\n    # Computing best number of iterations on an internal validation set\n    XV_train, XV_valid, yv_train, yv_valid = train_test_split(XV, y_valid, test_size=0.15, random_state=random_state)\n    xgb.fit(XV_train, yv_train, eval_set=[(XV_valid, yv_valid)],\n            eval_metric='mlogloss',\n            early_stopping_rounds=15, verbose=False)\n    xgb.n_estimators = xgb.best_iteration\n    xgb.fit(XV, y_valid.ravel())\n    y_gb = xgb.predict_proba(XT)\n    print('{:20s} {:2s} {:1.7f}'.format('Gradient_Boost:', 'logloss =\u003e', log_loss(y_test, y_gb)))\n    ll_gb.append(log_loss(y_test, y_gb)) #Saving the logloss score\n    print('')\n\n    print(f'Log-Loss for Base Learners:')\n    print(f'{ll_sc}')\n\n    # Comparison of the 3L ENSEMBLE techniques (Ensemble1, Ensemble2, Ensemble3) with Ensemble Logistic \u0026 Ensemble XGBOOST (plotting the results)\n    #classes = 4\n\n    ll_sc = np.array(ll_sc).reshape(-1, len(clfs)).T\n    #print(ll_sc)\n    ll_ensemble1 = np.array(ll_ensemble1)\n    ll_ensemble2 = np.array(ll_ensemble2)\n    ll_ensemble3 = np.array(ll_ensemble3)\n    ll_ensemble1_cc = np.array(ll_ensemble1_cc)\n    ll_ensemble2_cc = np.array(ll_ensemble2_cc)\n    ll_lr = np.array(ll_lr)\n    ll_gb = np.array(ll_gb)\n\n\n    plt.figure(figsize=(10,10))\n    plt.plot(ll_sc, color='black', label='Single_Classifiers')\n\n    for i in range(1, 6):\n        plt.plot(ll_sc[i], color='black')\n    plt.title('Log-loss of the different models.')\n    plt.xlabel('Testing on LDIGITS DATASET with only 4 classes')\n    plt.ylabel('Log-loss')\n    plt.grid(True)\n    plt.legend(loc=1)\n    plt.show()\n\n    plt.title('Log-loss of the different models.')\n    plt.xlabel('Testing on DIGITS DATASET with only 4 classes')\n    plt.ylabel('Log-loss')\n    plt.plot(ll_lr, 'bo-', label='EN_LogisticRegression', )\n    plt.plot(ll_gb, 'mo-', label='EN_XGBoost')\n    plt.plot(ll_ensemble1, 'yo-', label='Ensemble1')\n    plt.plot(ll_ensemble1_cc, 'ko-', label='Calibrated Ensemble1')\n    plt.plot(ll_ensemble2, 'go-', label='Ensemble2')\n    plt.plot(ll_ensemble2_cc, 'ko-', label='Calibrated Ensemble2')\n    plt.plot(ll_ensemble3, 'ro-', label='Ensemble_3rd_layer')\n\n    plt.grid(True)\n    plt.legend(loc=1)\n    plt.show()\n\n    print(np.argmax(y_thirdlayer, axis=1))\n    y_pred_meta_self_learner = np.argmax(y_thirdlayer, axis=1)\n\n    print(f'Predictions from the final layer:')\n    print(f'{y_pred_meta_self_learner}')\n\n    #Get the confusion matrix\n    cf_matrix = confusion_matrix(y_test, y_pred_meta_self_learner)\n    print(cf_matrix)\n    make_confusion_matrix(cf_matrix, figsize=(8,6), cbar=False, title='Confusion Matrix')\n\n    # Visualisation of plots\n    mc = MultiClassClassification(y_test, y_thirdlayer, labels=[0, 1, 2, 3])\n    plt.figure(figsize=(13,4))\n    plt.subplot(131)\n    mc.plot_roc()\n    plt.subplot(132)\n    mc.plot_confusion_matrix()\n    plt.subplot(133)\n    mc.plot_confusion_matrix(normalize=True)\n\n    plt.savefig('figures/images/plot_multi_classification.png')\n    plt.show()\n\n    mc.print_report()\n\n.. image:: figures/images/custom_CM.png\n\n.. image:: figures/images/results.png\n\n.. image:: figures/images/Testdata-Log-loss-plot-meta-self-learner.png\n\n.. image:: figures/images/Testdata-Log-loss-plot-individual-classifiers.png\n\nFull Demo\n=========\n## Important Links\n------------------\n- Find the full Notebook for the above illustration here : https://github.com/ajayarunachalam/meta-self-learner/blob/main/Full_Demo_Tested.ipynb\n\nTo Launch Above Illustration on Terminal\n----------------------------------------\nUsing Terminal:\n\n.. code:: sh\n\n    python tested_example.py\n\n\nLicense\n=======\nCopyright 2021-2022 Ajay Arunachalam \u003cajay.arunachalam08@gmail.com\u003e\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. © 2021 GitHub, Inc.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fajayarunachalam%2Fmeta-self-learner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fajayarunachalam%2Fmeta-self-learner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fajayarunachalam%2Fmeta-self-learner/lists"}