{"id":13688841,"url":"https://github.com/MSD-IRIMAS/Multi_Comparison_Matrix","last_synced_at":"2025-05-01T20:30:59.836Z","repository":{"id":162971629,"uuid":"610158277","full_name":"MSD-IRIMAS/Multi_Comparison_Matrix","owner":"MSD-IRIMAS","description":"Multi Comparison Matrix: A long term approach to benchmark evaluations","archived":false,"fork":false,"pushed_at":"2025-04-30T14:59:31.000Z","size":1436,"stargazers_count":21,"open_issues_count":1,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-30T15:57:00.718Z","etag":null,"topics":["benchmarking","critical-difference-diagram","evaluation","timeseriesclassification"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MSD-IRIMAS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-06T08:00:02.000Z","updated_at":"2025-04-30T14:48:03.000Z","dependencies_parsed_at":"2024-04-24T14:00:44.827Z","dependency_job_id":"078ccd01-d1dc-4eef-a6bc-3bd769b415be","html_url":"https://github.com/MSD-IRIMAS/Multi_Comparison_Matrix","commit_stats":null,"previous_names":["msd-irimas/multi_pairwise_comparison"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MSD-IRIMAS%2FMulti_Comparison_Matrix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MSD-IRIMAS%2FMulti_Comparison_Matrix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MSD-IRIMAS%2FMulti_Comparison_Matrix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MSD-IRIMAS%2FMulti_Comparison_Matrix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MSD-IRIMAS","download_url":"https://codeload.github.com/MSD-IRIMAS/Multi_Comparison_Matrix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251940537,"owners_count":21668554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","critical-difference-diagram","evaluation","timeseriesclassification"],"created_at":"2024-08-02T15:01:24.543Z","updated_at":"2025-05-01T20:30:59.829Z","avatar_url":"https://github.com/MSD-IRIMAS.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Multi-Comparison Matrix (MCM)\n\n### This repository contains the software for our paper titled \"[An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set](https://arxiv.org/abs/2305.11921)\". This work has been done by [Ali Ismail-Fawaz](https://hadifawaz1999.github.io/), [Angus Dempster](https://dblp.uni-trier.de/pid/251/8985.html), [Chang Wei Tan](https://changweitan.com/), [Matthieu Herrmann](https://orcid.org/0000-0002-0074-470X), [Lynn Miller](https://au.linkedin.com/in/lynn-miller-bb1aa539), [Daniel Schmidt](https://research.monash.edu/en/persons/daniel-schmidt), [Stefano Berretti](http://www.micc.unifi.it/berretti/), [Jonathan Weber](https://www.jonathan-weber.eu/), [Maxime Devanne](https://maxime-devanne.com/), [Germain Forestier](https://germain-forestier.info/) and [Geoff I. Webb](https://i.giwebb.com/).\n\n## Can now be used through [PyPl](https://pypi.org/project/multi-comp-matrix/)\n\nSimply install using `pip install multi-comp-matrix` and use it as explained in the below example section\n\n## Papers Using the MCM:\n\n1. Middlehurst et al. 2024 \"[Bake off redux: a review and experimental evaluation of recent time series classification algorithms](https://link.springer.com/article/10.1007/s10618-024-01022-1)\" Data Mining and Knowledge Discovery\n2. Ismail-Fawaz et al. 2024 \"[Finding foundation models for time series classification with a pretext task](https://doi.org/10.1007/978-981-97-2650-9_10)\" The Pacific-Asia Conference on Knowledge Discovery and Data Mining - International Workshop on Temporal Analytics\n3. Foumani et al. 2023 \"[Series2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classification](https://doi.org/10.1007/s10618-024-01043-w)\" Data Mining and Knowledge Discovery\n4. Holder et al. 2023 \"[A review and evaluation of elastic distance functions for time series clustering]([A review and evaluation of elastic distance functions for time series clustering](https://link.springer.com/article/10.1007/s10115-023-01952-0))\" Knowledge and Information Systems\n5. Ismail-Fawaz et al. 2023 \"[LITE: Light Inception with boosTing tEchniques for Time Series Classification](https://ieeexplore.ieee.org/abstract/document/10302569)\" IEEE 10th International Conference on Data Science and Advanced Analytics\n6. Koh et al. 2023 \"[PSICHIC: physicochemical graph neural network for learning protein-ligand interaction fingerprints from sequence data](https://www.biorxiv.org/content/10.1101/2023.09.17.558145v1.abstract)\" bioRxiv\n7. Ayllón-Gavilán et al. 2023 \"[Convolutional and Deep Learning based techniques for Time Series Ordinal Classification](https://arxiv.org/abs/2306.10084)\"\n8. Ismail-Fawaz et al. 2023 \"[ShapeDBA: Generating Effective Time Series Prototypes Using ShapeDTW Barycenter Averaging](https://link.springer.com/chapter/10.1007/978-3-031-49896-1_9)\" The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases Workshop on Advanced Analytics and Learning on Temporal Data\n9. Dempster et al. 2023 \"[QUANT: A Minimalist Interval Method for Time Series Classification](https://doi.org/10.1007/s10618-024-01036-9)\" Data Mining and Knowledge Discovery\n10. Holder et al. 2023 \"[Clustering Time Series with k-Medoids Based Algorithms](https://link.springer.com/chapter/10.1007/978-3-031-49896-1_4)\" The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases Workshop on Advanced Analytics and Learning on Temporal Data\n11. Guijo-Rubio et al. 2023 \"[Unsupervised feature based algorithms for time series extrinsic regression](https://doi.org/10.1007/s10618-024-01027-w)\" Data Mining and Knowledge Discovery\n12. Fischer et al. 2024 \"[Towards more sustainable and trustworthy reporting in machine learning](https://doi.org/10.1007/s10618-024-01020-3).\" Data Mining and Knowledge Discovery\n13. Middlehurst et al. 2024 \"[aeon: a Python toolkit for learning from time series](https://www.jmlr.org/papers/v25/23-1444.html).\"\n14. da Silva et al. 2024 \"[Artist Similarity based on Heterogeneous Graph Neural Networks](https://doi.org/10.1109/TASLP.2024.3437170).\" IEEE/ACM Transactions on Audio, Speech, and Language Processing\n15. Renault, Aurélien, et al. [\"Early Classification of Time Series: Taxonomy and Benchmark.\"](https://arxiv.org/abs/2406.18332) arXiv preprint arXiv:2406.18332 (2024).\n16. Spinnato, Francesco, et al. [\"Fast, Interpretable and Deterministic Time Series Classification with a Bag-Of-Receptive-Fields.\"](https://link.springer.com/article/10.1007/s10618-024-01020-3) Data Mining and Knowledge Discovery (2024).\n17. Lo, Mouhamadou Mansour, et al. [\"Time series classification with random convolution kernels based transforms: pooling operators and input representations matter.\"](https://arxiv.org/abs/2409.01115) arXiv preprint arXiv:2409.01115 (2024).\n\n## Summary\n\nThis repo is a long term used benchmark method that generates a Multi-Comparison Matrix where the user ca choose whether to include a full pairwise multi-comparate comparison or to choose which ones to be included or excluded in the rows and columns of the matrix.\n\n## Input Format\n\nThe input format is in a ```.csv``` file containing the statistics of each classifiers as the format of [this example](https://github.com/MSD-IRIMAS/Multi_Pairwise_Comparison/blob/main/results_example.csv).\n\n## Usage of Code - Plot the MCM\n\nIn order for the user to plot the MCM, first thing is to load the ```.csv``` file into a ```pandas``` dataframe and feed it to the ```compare``` function. The user should specify the ```pdf_savename```, ```png_savename```, ```csv_savename``` or ```tex_savename``` parameter in order to save the output figure in ```pdf```, ```png```, ```csv``` or ```tex``` formats respecively.\n\n## Examples\n\nGenerating the MCM on the [following example](https://github.com/MSD-IRIMAS/Multi_Pairwise_Comparison/blob/main/results_example.csv) produces the following. To generate the following figure, the user follows this simple code:\n\n```\nimport pandas as pd\nfrom multi_comp_matrix import MCM\n\ndf_results = pd.read_csv('path/to/csv')\n\noutput_dir = '/output/directory/desired'\n\nMCM.compare(\n        output_dir=output_dir,\n        df_results=df_results,\n        pdf_savename=\"heatmap\",\n        png_savename=\"heatmap\",\n    )\n```\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"heatmap.png\" alt=\"heatmap-example\"/\u003e\n\u003c/p\u003e\n\nGenerating the MCM on the [following example](https://github.com/MSD-IRIMAS/Multi_Pairwise_Comparison/blob/main/results_example.csv) by excluding ```clf1``` and ```clf3``` from the columns.\n\n```\nimport pandas as pd\nfrom multi_comp_matrix import MCM\n\ndf_results = pd.read_csv('path/to/csv')\n\noutput_dir = '/output/directory/desired'\n\nMCM.compare(\n        output_dir=output_dir,\n        df_results=df_results,\n        excluded_col_comparates=['clf1','clf3'],\n        png_savename='heatline_vertical',\n        tex_savename='heatline_vertical',\n        include_ProbaWinTieLoss=True\n    )\n```\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"heatline_vertical.png\" alt=\"heatline-vertical-example\"/\u003e\n\u003c/p\u003e\n\nand by excluding them in the rows.\n\n```\nimport pandas as pd\nfrom multi_comp_matrix import MCM\n\ndf_results = pd.read_csv('path/to/csv')\n\noutput_dir = '/output/directory/desired'\n\nMCM.compare(\n        output_dir=output_dir,\n        df_results=df_results,\n        excluded_row_comparates=['clf1','clf3'],\n        png_savename='heatline_horizontal',\n        csv_savename='heatline_horizontal',\n    )\n```\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003cimg src=\"heatline_horizontal.png\" alt=\"heatline-horizontal-example\"/\u003e\n\u003c/p\u003e\n\n## Colormap\n\nThe colormap can be [customized](https://github.com/MSD-IRIMAS/Multi_Comparison_Matrix/blob/ce2c4ebfeda57be5db56e96e71d118996147c91e/multi_comp_matrix/MCM.py#L57) to suit different needs. Users can modify it to enhance readability, including options for colorblind-friendly palettes (e.g., viridis, cividis).\n\n## Requirements\n\nThe following python packages are required for the usage of the module:\n\n1. ```numpy==1.24.4```\n2. ```pandas==2.0.3```\n3. ```matplotlib==3.7.4```\n4. ```scipy==1.10.0```\n5. ```baycomp==1.0```\n6. ```tqdm==4.66.1```\n\n\n## Citation\n\nIf you use this work please make sure you cite this paper:\n```\n@article{ismail2023approach,\n  title={An Approach To Multiple Comparison Benchmark Evaluations That Is Stable Under Manipulation Of The Comparate Set},\n  author={Ismail-Fawaz, Ali and Dempster, Angus and Tan, Chang Wei and Herrmann, Matthieu and Miller, Lynn and Schmidt, Daniel F and Berretti, Stefano and Weber, Jonathan and Devanne, Maxime and Forestier, Germain and Webb, Geoff I},\n  journal={arXiv preprint arXiv:2305.11921},\n  year={2023}\n}\n```\n\n## Acknowledgments\n\nThe work reported in this paper has been supported by the Australian Research Council\nunder grant DP210100072; the ANR TIMES project (grant ANR-17- CE23-0015); and ANR\nDELEGATION project (grant ANR-21-CE23-0014) of the French Agence Nationale de la\nRecherche. The authors would like to thank Professor Eamonn Keogh and all the people\nwho have contributed to the UCR time series classification archive.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMSD-IRIMAS%2FMulti_Comparison_Matrix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMSD-IRIMAS%2FMulti_Comparison_Matrix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMSD-IRIMAS%2FMulti_Comparison_Matrix/lists"}