{"id":13712622,"url":"https://github.com/yzhao062/combo","last_synced_at":"2025-04-08T09:11:28.029Z","repository":{"id":53488171,"uuid":"196780423","full_name":"yzhao062/combo","owner":"yzhao062","description":"(AAAI' 20) A Python Toolbox for Machine Learning Model Combination","archived":false,"fork":false,"pushed_at":"2023-01-14T04:46:24.000Z","size":5189,"stargazers_count":648,"open_issues_count":15,"forks_count":106,"subscribers_count":29,"default_branch":"master","last_synced_at":"2025-04-01T05:33:36.056Z","etag":null,"topics":["aggregation","data-mining","data-science","ensemble-learning","machine-learning","machine-learning-pipelines","model-combination","pipeline-framework","python"],"latest_commit_sha":null,"homepage":"https://pycombo.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yzhao062.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGES.txt","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-14T01:13:36.000Z","updated_at":"2025-03-24T15:12:07.000Z","dependencies_parsed_at":"2023-02-09T18:46:23.804Z","dependency_job_id":null,"html_url":"https://github.com/yzhao062/combo","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yzhao062%2Fcombo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yzhao062%2Fcombo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yzhao062%2Fcombo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yzhao062%2Fcombo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yzhao062","download_url":"https://codeload.github.com/yzhao062/combo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247809964,"owners_count":20999816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aggregation","data-mining","data-science","ensemble-learning","machine-learning","machine-learning-pipelines","model-combination","pipeline-framework","python"],"created_at":"2024-08-02T23:01:20.545Z","updated_at":"2025-04-08T09:11:27.998Z","avatar_url":"https://github.com/yzhao062.png","language":"Python","readme":"combo: A Python Toolbox for Machine Learning Model Combination\n==============================================================\n\n\n**Deployment \u0026 Documentation \u0026 Stats**\n\n.. image:: https://img.shields.io/pypi/v/combo.svg?color=brightgreen\n   :target: https://pypi.org/project/combo/\n   :alt: PyPI version\n\n\n.. image:: https://readthedocs.org/projects/pycombo/badge/?version=latest\n   :target: https://pycombo.readthedocs.io/en/latest/?badge=latest\n   :alt: Documentation Status\n\n\n.. image:: https://mybinder.org/badge_logo.svg\n   :target: https://mybinder.org/v2/gh/yzhao062/combo/master\n   :alt: Binder\n\n\n.. image:: https://img.shields.io/github/stars/yzhao062/combo.svg\n   :target: https://github.com/yzhao062/combo/stargazers\n   :alt: GitHub stars\n\n\n.. image:: https://img.shields.io/github/forks/yzhao062/combo.svg?color=blue\n   :target: https://github.com/yzhao062/combo/network\n   :alt: GitHub forks\n\n\n.. image:: https://pepy.tech/badge/combo\n   :target: https://pepy.tech/project/combo\n   :alt: Downloads\n\n\n.. image:: https://pepy.tech/badge/combo/month\n   :target: https://pepy.tech/project/combo\n   :alt: Downloads\n\n\n----\n\n\n**Build Status \u0026 Coverage \u0026 Maintainability \u0026 License**\n\n.. image:: https://github.com/yzhao062/combo/actions/workflows/testing.yml/badge.svg\n   :target: https://github.com/yzhao062/combo/actions/workflows/testing.yml\n   :alt: testing\n\n\n.. image:: https://circleci.com/gh/yzhao062/combo.svg?style=svg\n   :target: https://circleci.com/gh/yzhao062/combo\n   :alt: Circle CI\n\n\n.. image:: https://ci.appveyor.com/api/projects/status/te7uieha87305ike/branch/master?svg=true\n   :target: https://ci.appveyor.com/project/yzhao062/combo/branch/master\n   :alt: Build status\n\n\n.. image:: https://coveralls.io/repos/github/yzhao062/combo/badge.svg\n   :target: https://coveralls.io/github/yzhao062/combo\n   :alt: Coverage Status\n\n\n.. image:: https://api.codeclimate.com/v1/badges/465ebba81e990abb357b/maintainability\n   :target: https://codeclimate.com/github/yzhao062/combo/maintainability\n   :alt: Maintainability\n\n\n.. image:: https://img.shields.io/github/license/yzhao062/combo.svg\n   :target: https://github.com/yzhao062/combo/blob/master/LICENSE\n   :alt: License\n\n\n----\n\n\n**combo** is a comprehensive Python toolbox for **combining machine learning (ML) models and scores**.\n**Model combination** can be considered as a subtask of `ensemble learning \u003chttps://en.wikipedia.org/wiki/Ensemble_learning\u003e`_,\nand has been widely used in real-world tasks and data science competitions like Kaggle [#Bell2007Lessons]_.\n**combo** has been used/introduced in various research works since its inception [#Raschka2020Machine]_ [#Zhao2019PyOD]_.\n\n**combo** library supports the combination of models and score from\nkey ML libraries such as `scikit-learn \u003chttps://scikit-learn.org/stable/index.html\u003e`_,\n`xgboost \u003chttps://xgboost.ai/\u003e`_, and `LightGBM \u003chttps://github.com/microsoft/LightGBM\u003e`_,\nfor crucial tasks including classification, clustering, anomaly detection.\nSee figure below for some representative combination approaches.\n\n.. image:: https://raw.githubusercontent.com/yzhao062/combo/master/docs/figs/framework_demo.png\n   :target: https://raw.githubusercontent.com/yzhao062/combo/master/docs/figs/framework_demo.png\n   :alt: Combination Framework Demo\n\n\n**combo** is featured for:\n\n* **Unified APIs, detailed documentation, and interactive examples** across various algorithms.\n* **Advanced and latest models**, such as Stacking/DCS/DES/EAC/LSCP.\n* **Comprehensive coverage** for classification, clustering, anomaly detection, and raw score.\n* **Optimized performance with JIT and parallelization** when possible, using `numba \u003chttps://github.com/numba/numba\u003e`_ and `joblib \u003chttps://github.com/joblib/joblib\u003e`_.\n\n\n**API Demo**\\ :\n\n.. code-block:: python\n\n\n   from combo.models.classifier_stacking import Stacking\n   # initialize a group of base classifiers\n   classifiers = [DecisionTreeClassifier(), LogisticRegression(),\n                  KNeighborsClassifier(), RandomForestClassifier(),\n                  GradientBoostingClassifier()]\n\n   clf = Stacking(base_estimators=classifiers) # initialize a Stacking model\n   clf.fit(X_train, y_train) # fit the model\n\n   # predict on unseen data\n   y_test_labels = clf.predict(X_test)  # label prediction\n   y_test_proba = clf.predict_proba(X_test)  # probability prediction\n\n\n**Citing combo**\\ :\n\n`combo paper \u003chttp://www.andrew.cmu.edu/user/yuezhao2/papers/20-aaai-combo.pdf\u003e`_ is published in\n`AAAI 2020 \u003chttps://aaai.org/Conferences/AAAI-20/\u003e`_ (demo track).\nIf you use combo in a scientific publication, we would appreciate citations to the following paper::\n\n    @inproceedings{zhao2020combo,\n      title={Combining Machine Learning Models and Scores using combo library},\n      author={Zhao, Yue and Wang, Xuejian and Cheng, Cheng and Ding, Xueying},\n      booktitle={Thirty-Fourth AAAI Conference on Artificial Intelligence},\n      month = {Feb},\n      year={2020},\n      address = {New York, USA}\n    }\n\nor::\n\n    Zhao, Y., Wang, X., Cheng, C. and Ding, X., 2020. Combining Machine Learning Models and Scores using combo library. Thirty-Fourth AAAI Conference on Artificial Intelligence.\n\n\n**Key Links and Resources**\\ :\n\n\n* `awesome-ensemble-learning \u003chttps://github.com/yzhao062/awesome-ensemble-learning\u003e`_ (ensemble learning related books, papers, and more)\n* `View the latest codes on Github \u003chttps://github.com/yzhao062/combo\u003e`_\n* `View the documentation \u0026 API \u003chttps://pycombo.readthedocs.io/\u003e`_\n* `View all examples \u003chttps://github.com/yzhao062/combo/tree/master/examples\u003e`_\n* `View the demo video for AAAI 2020 \u003chttps://youtu.be/PaSJ49Ij7w4\u003e`_\n* `Execute Interactive Jupyter Notebooks \u003chttps://mybinder.org/v2/gh/yzhao062/combo/master\u003e`_\n\n\n**Table of Contents**\\ :\n\n\n* `Installation \u003c#installation\u003e`_\n* `API Cheatsheet \u0026 Reference \u003c#api-cheatsheet--reference\u003e`_\n* `Implemented Algorithms \u003c#implemented-algorithms\u003e`_\n* `Example 1: Classifier Combination with Stacking/DCS/DES \u003c#example-of-stackingdcsdes\u003e`_\n* `Example 2: Simple Classifier Combination \u003c#example-of-classifier-combination\u003e`_\n* `Example 3: Clustering Combination \u003c#example-of-clustering-combination\u003e`_\n* `Example 4: Outlier Detector Combination \u003c#example-of-outlier-detector-combination\u003e`_\n* `Development Status \u003c#development-status\u003e`_\n* `Inclusion Criteria \u003c#inclusion-criteria\u003e`_\n\n\n----\n\n\nInstallation\n^^^^^^^^^^^^\n\nIt is recommended to use **pip** for installation. Please make sure\n**the latest version** is installed, as combo is updated frequently:\n\n.. code-block:: bash\n\n   pip install combo            # normal install\n   pip install --upgrade combo  # or update if needed\n   pip install --pre combo      # or include pre-release version for new features\n\nAlternatively, you could clone and run setup.py file:\n\n.. code-block:: bash\n\n   git clone https://github.com/yzhao062/combo.git\n   cd combo\n   pip install .\n\n\n**Required Dependencies**\\ :\n\n\n* Python 3.5, 3.6, or 3.7\n* joblib\n* matplotlib (**optional for running examples**)\n* numpy\u003e=1.13\n* numba\u003e=0.35\n* pyod\n* scipy\u003e=0.19.1\n* scikit_learn\u003e=0.20\n\n\n**Note on Python 2**\\ :\nThe maintenance of Python 2.7 will be stopped by January 1, 2020 (see `official announcement \u003chttps://github.com/python/devguide/pull/344\u003e`_).\nTo be consistent with the Python change and combo's dependent libraries, e.g., scikit-learn,\n**combo only supports Python 3.5+** and we encourage you to use\nPython 3.5 or newer for the latest functions and bug fixes. More information can\nbe found at `Moving to require Python 3 \u003chttps://python3statement.org/\u003e`_.\n\n\n----\n\n\nAPI Cheatsheet \u0026 Reference\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nFull API Reference: (https://pycombo.readthedocs.io/en/latest/api.html).\nThe following APIs are consistent for most of the models\n(API Cheatsheet: https://pycombo.readthedocs.io/en/latest/api_cc.html).\n\n* **fit(X, y)**\\ : Fit estimator. y is optional for unsupervised methods.\n* **predict(X)**\\ : Predict on a particular sample once the estimator is fitted.\n* **predict_proba(X)**\\ : Predict the probability of a sample belonging to each class once the estimator is fitted.\n* **fit_predict(X, y)**\\ : Fit estimator and predict on X. y is optional for unsupervised methods.\n\nFor raw score combination (after the score matrix is generated),\nuse individual methods from\n`\"score_comb.py\" \u003chttps://github.com/yzhao062/combo/blob/master/combo/models/score_comb.py\u003e`_ directly.\nRaw score combination API: (https://pycombo.readthedocs.io/en/latest/api.html#score-combination).\n\n\n----\n\n\nImplemented Algorithms\n^^^^^^^^^^^^^^^^^^^^^^\n\n**combo** groups combination frameworks by tasks. General purpose methods are\nfundamental ones which can be applied to various tasks.\n\n===================  ======================================================================================================  =====  ===========================================\nTask                 Algorithm                                                                                               Year   Ref\n===================  ======================================================================================================  =====  ===========================================\nGeneral Purpose      Average \u0026 Weighted Average: average across all scores/prediction results, maybe with weights            N/A    [#Zhou2012Ensemble]_\nGeneral Purpose      Maximization: simple combination by taking the maximum scores                                           N/A    [#Zhou2012Ensemble]_\nGeneral Purpose      Median: take the median value across all scores/prediction results                                      N/A    [#Zhou2012Ensemble]_\nGeneral Purpose      Majority Vote \u0026 Weighted Majority Vote                                                                  N/A    [#Zhou2012Ensemble]_\nClassification       SimpleClassifierAggregator: combining classifiers by general purpose methods above                      N/A    N/A\nClassification       DCS: Dynamic Classifier Selection (Combination of multiple classifiers using local accuracy estimates)  1997   [#Woods1997Combination]_\nClassification       DES: Dynamic Ensemble Selection (From dynamic classifier selection to dynamic ensemble selection)       2008   [#Ko2008From]_\nClassification       Stacking (meta ensembling): use a meta learner to learn the base classifier results                     N/A    [#Gorman2016Kaggle]_\nClustering           Clusterer Ensemble: combine the results of multiple clustering results by relabeling                    2006   [#Zhou2006Clusterer]_\nClustering           Combining multiple clusterings using evidence accumulation (EAC)                                        2002   [#Fred2005Combining]_\nAnomaly Detection    SimpleDetectorCombination: combining outlier detectors by general purpose methods above                 N/A    [#Aggarwal2017Outlier]_\nAnomaly Detection    Average of Maximum (AOM): divide base detectors into subgroups to take the maximum, and then average    2015   [#Aggarwal2015Theoretical]_\nAnomaly Detection    Maximum of Average (MOA): divide base detectors into subgroups to take the average, and then maximize   2015   [#Aggarwal2015Theoretical]_\nAnomaly Detection    XGBOD: a semi-supervised combination framework for outlier detection                                    2018   [#Zhao2018XGBOD]_\nAnomaly Detection    Locally Selective Combination (LSCP)                                                                    2019   [#Zhao2019LSCP]_\n===================  ======================================================================================================  =====  ===========================================\n\n\n**The comparison among selected implemented models** is made available below\n(\\ `Figure \u003chttps://raw.githubusercontent.com/yzhao062/combo/master/examples/compare_selected_classifiers.png\u003e`_\\ ,\n`compare_selected_classifiers.py \u003chttps://github.com/yzhao062/combo/blob/master/examples/compare_selected_classifiers.py\u003e`_\\, `Interactive Jupyter Notebooks \u003chttps://mybinder.org/v2/gh/yzhao062/combo/master\u003e`_\\ ).\nFor Jupyter Notebooks, please navigate to **\"/notebooks/compare_selected_classifiers.ipynb\"**.\n\n\n.. image:: https://raw.githubusercontent.com/yzhao062/combo/master/examples/compare_selected_classifiers.png\n   :target: https://raw.githubusercontent.com/yzhao062/combo/master/examples/compare_selected_classifiers.png\n   :alt: Comparison of Selected Models\n\n\n----\n\n\n**All implemented modes** are associated with examples, check\n`\"combo examples\" \u003chttps://github.com/yzhao062/combo/blob/master/examples\u003e`_\nfor more information.\n\n\nExample of Stacking/DCS/DES\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n\n`\"examples/classifier_stacking_example.py\" \u003chttps://github.com/yzhao062/combo/blob/master/examples/classifier_stacking_example.py\u003e`_\ndemonstrates the basic API of stacking (meta ensembling). `\"examples/classifier_dcs_la_example.py\" \u003chttps://github.com/yzhao062/combo/blob/master/examples/classifier_dcs_la_example.py\u003e`_\ndemonstrates the basic API of Dynamic Classifier Selection by Local Accuracy. `\"examples/classifier_des_la_example.py\" \u003chttps://github.com/yzhao062/combo/blob/master/examples/classifier_des_la_example.py\u003e`_\ndemonstrates the basic API of Dynamic Ensemble Selection by Local Accuracy.\n\nIt is noted **the basic API is consistent across all these models**.\n\n\n#. Initialize a group of classifiers as base estimators\n\n   .. code-block:: python\n\n\n      # initialize a group of classifiers\n      classifiers = [DecisionTreeClassifier(random_state=random_state),\n                     LogisticRegression(random_state=random_state),\n                     KNeighborsClassifier(),\n                     RandomForestClassifier(random_state=random_state),\n                     GradientBoostingClassifier(random_state=random_state)]\n\n\n#. Initialize, fit, predict, and evaluate with Stacking\n\n   .. code-block:: python\n\n\n      from combo.models.classifier_stacking import Stacking\n\n      clf = Stacking(base_estimators=classifiers, n_folds=4, shuffle_data=False,\n                   keep_original=True, use_proba=False, random_state=random_state)\n\n      clf.fit(X_train, y_train)\n      y_test_predict = clf.predict(X_test)\n      evaluate_print('Stacking | ', y_test, y_test_predict)\n\n\n#. See a sample output of classifier_stacking_example.py\n\n   .. code-block:: bash\n\n\n      Decision Tree        | Accuracy:0.9386, ROC:0.9383, F1:0.9521\n      Logistic Regression  | Accuracy:0.9649, ROC:0.9615, F1:0.973\n      K Neighbors          | Accuracy:0.9561, ROC:0.9519, F1:0.9662\n      Gradient Boosting    | Accuracy:0.9605, ROC:0.9524, F1:0.9699\n      Random Forest        | Accuracy:0.9605, ROC:0.961, F1:0.9693\n\n      Stacking             | Accuracy:0.9868, ROC:0.9841, F1:0.9899\n\n\n----\n\n\nExample of Classifier Combination\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n\n`\"examples/classifier_comb_example.py\" \u003chttps://github.com/yzhao062/combo/blob/master/examples/classifier_comb_example.py\u003e`_\ndemonstrates the basic API of predicting with multiple classifiers. **It is noted that the API across all other algorithms are consistent/similar**.\n\n#. Initialize a group of classifiers as base estimators\n\n   .. code-block:: python\n\n\n      # initialize a group of classifiers\n      classifiers = [DecisionTreeClassifier(random_state=random_state),\n                     LogisticRegression(random_state=random_state),\n                     KNeighborsClassifier(),\n                     RandomForestClassifier(random_state=random_state),\n                     GradientBoostingClassifier(random_state=random_state)]\n\n\n#. Initialize, fit, predict, and evaluate with a simple aggregator (average)\n\n   .. code-block:: python\n\n\n      from combo.models.classifier_comb import SimpleClassifierAggregator\n\n      clf = SimpleClassifierAggregator(classifiers, method='average')\n      clf.fit(X_train, y_train)\n      y_test_predicted = clf.predict(X_test)\n      evaluate_print('Combination by avg   |', y_test, y_test_predicted)\n\n\n\n#. See a sample output of classifier_comb_example.py\n\n   .. code-block:: bash\n\n\n      Decision Tree        | Accuracy:0.9386, ROC:0.9383, F1:0.9521\n      Logistic Regression  | Accuracy:0.9649, ROC:0.9615, F1:0.973\n      K Neighbors          | Accuracy:0.9561, ROC:0.9519, F1:0.9662\n      Gradient Boosting    | Accuracy:0.9605, ROC:0.9524, F1:0.9699\n      Random Forest        | Accuracy:0.9605, ROC:0.961, F1:0.9693\n\n      Combination by avg   | Accuracy:0.9693, ROC:0.9677, F1:0.9763\n      Combination by w_avg | Accuracy:0.9781, ROC:0.9716, F1:0.9833\n      Combination by max   | Accuracy:0.9518, ROC:0.9312, F1:0.9642\n      Combination by w_vote| Accuracy:0.9649, ROC:0.9644, F1:0.9728\n      Combination by median| Accuracy:0.9693, ROC:0.9677, F1:0.9763\n\n\n----\n\n\nExample of Clustering Combination\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n\n`\"examples/cluster_comb_example.py\" \u003chttps://github.com/yzhao062/combo/blob/master/examples/cluster_comb_example.py\u003e`_\ndemonstrates the basic API of combining multiple base clustering estimators. `\"examples/cluster_eac_example.py\" \u003chttps://github.com/yzhao062/combo/blob/master/examples/cluster_eac_example.py\u003e`_\ndemonstrates the basic API of Combining multiple clusterings using evidence accumulation (EAC).\n\n#. Initialize a group of clustering methods as base estimators\n\n   .. code-block:: python\n\n\n      # Initialize a set of estimators\n      estimators = [KMeans(n_clusters=n_clusters),\n                    MiniBatchKMeans(n_clusters=n_clusters),\n                    AgglomerativeClustering(n_clusters=n_clusters)]\n\n\n#. Initialize a Clusterer Ensemble class and fit the model\n\n   .. code-block:: python\n\n\n      from combo.models.cluster_comb import ClustererEnsemble\n      # combine by Clusterer Ensemble\n      clf = ClustererEnsemble(estimators, n_clusters=n_clusters)\n      clf.fit(X)\n\n\n#. Get the aligned results\n\n   .. code-block:: python\n\n\n      # generate the labels on X\n      aligned_labels = clf.aligned_labels_\n      predicted_labels = clf.labels_\n\n\n\nExample of Outlier Detector Combination\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n\n`\"examples/detector_comb_example.py\" \u003chttps://github.com/yzhao062/combo/blob/master/examples/detector_comb_example.py\u003e`_\ndemonstrates the basic API of combining multiple base outlier detectors.\n\n#. Initialize a group of outlier detection methods as base estimators\n\n   .. code-block:: python\n\n\n      # Initialize a set of estimators\n      detectors = [KNN(), LOF(), OCSVM()]\n\n\n#. Initialize a simple averaging aggregator, fit the model, and make\n   the prediction.\n\n   .. code-block:: python\n\n\n      from combo.models.detector combination import SimpleDetectorAggregator\n      clf = SimpleDetectorAggregator(base_estimators=detectors)\n      clf_name = 'Aggregation by Averaging'\n      clf.fit(X_train)\n\n      y_train_pred = clf.labels_  # binary labels (0: inliers, 1: outliers)\n      y_train_scores = clf.decision_scores_  # raw outlier scores\n\n      # get the prediction on the test data\n      y_test_pred = clf.predict(X_test)  # outlier labels (0 or 1)\n      y_test_scores = clf.decision_function(X_test)  # outlier scores\n\n\n#. Evaluate the prediction using ROC and Precision @ Rank n.\n\n   .. code-block:: python\n\n      # evaluate and print the results\n      print(\"\\nOn Training Data:\")\n      evaluate_print(clf_name, y_train, y_train_scores)\n      print(\"\\nOn Test Data:\")\n      evaluate_print(clf_name, y_test, y_test_scores)\n\n#. See sample outputs on both training and test data.\n\n   .. code-block:: bash\n\n      On Training Data:\n      Aggregation by Averaging ROC:0.9994, precision @ rank n:0.95\n\n      On Test Data:\n      Aggregation by Averaging ROC:1.0, precision @ rank n:1.0\n\n\n----\n\n\nDevelopment Status\n^^^^^^^^^^^^^^^^^^\n\n**combo** is currently **under development** as of Feb, 2020. A concrete plan has\nbeen laid out and will be implemented in the next few months.\n\nSimilar to other libraries built by us, e.g., Python Outlier Detection Toolbox\n(`pyod \u003chttps://github.com/yzhao062/pyod\u003e`_),\n**combo** is also targeted to be published in *Journal of Machine Learning Research (JMLR)*,\n`open-source software track \u003chttp://www.jmlr.org/mloss/\u003e`_. A demo paper has been presented in\n*AAAI 2020* for progress update.\n\n**Watch \u0026 Star** to get the latest update! Also feel free to send me an email (zhaoy@cmu.edu)\nfor suggestions and ideas.\n\n\n----\n\n\nInclusion Criteria\n^^^^^^^^^^^^^^^^^^\n\nSimilarly to scikit-learn, We mainly consider well-established algorithms for inclusion.\nA rule of thumb is at least two years since publication, 50+ citations, and usefulness.\n\nHowever, we encourage the author(s) of newly proposed models to share and add your implementation into combo\nfor boosting ML accessibility and reproducibility.\nThis exception only applies if you could commit to the maintenance of your model for at least two year period.\n\n\n----\n\n\nReference\n^^^^^^^^^\n\n.. [#Aggarwal2015Theoretical] Aggarwal, C.C. and Sathe, S., 2015. Theoretical foundations and algorithms for outlier ensembles. *ACM SIGKDD Explorations Newsletter*, 17(1), pp.24-47.\n\n.. [#Aggarwal2017Outlier] Aggarwal, C.C. and Sathe, S., 2017. Outlier ensembles: An introduction. Springer.\n\n.. [#Bell2007Lessons] Bell, R.M. and Koren, Y., 2007. Lessons from the Netflix prize challenge. *SIGKDD Explorations*, 9(2), pp.75-79.\n\n.. [#Gorman2016Kaggle] Gorman, B. (2016). A Kaggler's Guide to Model Stacking in Practice. [online] The Official Blog of Kaggle.com. Available at: http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice [Accessed 26 Jul. 2019].\n\n.. [#Ko2008From] Ko, A.H., Sabourin, R. and Britto Jr, A.S., 2008. From dynamic classifier selection to dynamic ensemble selection. *Pattern recognition*, 41(5), pp.1718-1731.\n\n.. [#Fred2005Combining] Fred, A. L. N., \u0026 Jain, A. K. (2005). Combining multiple clusterings using evidence accumulation. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 27(6), 835–850. https://doi.org/10.1109/TPAMI.2005.113\n\n.. [#Raschka2020Machine] Raschka, S., Patterson, J. and Nolet, C., 2020. Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence. arXiv preprint arXiv:2002.04803.\n\n.. [#Woods1997Combination] Woods, K., Kegelmeyer, W.P. and Bowyer, K., 1997. Combination of multiple classifiers using local accuracy estimates. *IEEE transactions on pattern analysis and machine intelligence*, 19(4), pp.405-410.\n\n.. [#Zhao2018XGBOD] Zhao, Y. and Hryniewicki, M.K. XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning. *IEEE International Joint Conference on Neural Networks*, 2018.\n\n.. [#Zhao2019LSCP] Zhao, Y., Nasrullah, Z., Hryniewicki, M.K. and Li, Z., 2019, May. LSCP: Locally selective combination in parallel outlier ensembles. In *Proceedings of the 2019 SIAM International Conference on Data Mining (SDM)*, pp. 585-593. Society for Industrial and Applied Mathematics.\n\n.. [#Zhao2019PyOD] Zhao, Y., Nasrullah, Z. and Li, Z., 2019. PyOD: A Python Toolbox for Scalable Outlier Detection. *Journal of Machine Learning Research*, 20, pp.1-7.\n\n.. [#Zhou2006Clusterer] Zhou, Z.H. and Tang, W., 2006. Clusterer ensemble. *Knowledge-Based Systems*, 19(1), pp.77-83.\n\n.. [#Zhou2012Ensemble] Zhou, Z.H., 2012. Ensemble methods: foundations and algorithms. Chapman and Hall/CRC.","funding_links":[],"categories":["其他_机器学习与深度学习","Table of Contents","Python","Sklearn实用程序","Machine Learning Frameworks"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyzhao062%2Fcombo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyzhao062%2Fcombo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyzhao062%2Fcombo/lists"}