{"id":15628137,"url":"https://github.com/kaleidophon/deep-significance","last_synced_at":"2025-04-04T11:16:55.671Z","repository":{"id":37242494,"uuid":"341677886","full_name":"Kaleidophon/deep-significance","owner":"Kaleidophon","description":"Enabling easy statistical significance testing for deep neural networks. ","archived":false,"fork":false,"pushed_at":"2024-07-01T11:48:13.000Z","size":5846,"stargazers_count":331,"open_issues_count":0,"forks_count":19,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-28T10:11:19.137Z","etag":null,"topics":["compare-scores","deep-learning","deep-neural-networks","deeplearning","dl","hypothesis-testing","hypothesis-tests","machine-learning","machinelearning","ml","significance-testing","statistical-significance","statistical-significance-test"],"latest_commit_sha":null,"homepage":"https://deep-significance.rtfd.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Kaleidophon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-23T20:14:17.000Z","updated_at":"2025-03-27T05:46:00.000Z","dependencies_parsed_at":"2023-01-22T03:20:37.133Z","dependency_job_id":"d7c9affe-04cd-4f10-8f74-693ea4e482c5","html_url":"https://github.com/Kaleidophon/deep-significance","commit_stats":{"total_commits":190,"total_committers":3,"mean_commits":"63.333333333333336","dds":0.0368421052631579,"last_synced_commit":"0d5191b69b4cde876c8895a049532161672b0146"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kaleidophon%2Fdeep-significance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kaleidophon%2Fdeep-significance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kaleidophon%2Fdeep-significance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Kaleidophon%2Fdeep-significance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Kaleidophon","download_url":"https://codeload.github.com/Kaleidophon/deep-significance/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247166171,"owners_count":20894654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compare-scores","deep-learning","deep-neural-networks","deeplearning","dl","hypothesis-testing","hypothesis-tests","machine-learning","machinelearning","ml","significance-testing","statistical-significance","statistical-significance-test"],"created_at":"2024-10-03T10:21:08.644Z","updated_at":"2025-04-04T11:16:55.649Z","avatar_url":"https://github.com/Kaleidophon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# deep-significance: Easy and Better Significance Testing for Deep Neural Networks\n\n[![Build Status](https://travis-ci.com/Kaleidophon/deep-significance.svg?branch=main)]()\n[![Coverage Status](https://coveralls.io/repos/github/Kaleidophon/deep-significance/badge.svg?branch=main\u0026service=github)](https://coveralls.io/github/Kaleidophon/deep-significance?branch=main)\n[![Compatibility](https://img.shields.io/badge/python-v3.7-blue)]()\n[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black)\n[![DOI](https://zenodo.org/badge/341677886.svg)](https://zenodo.org/badge/latestdoi/341677886)\n\n![](img/logo.png)\n\n**Contents**\n\n* [:interrobang: Why](#interrobang-why)\n* [:inbox_tray: Installation](#inbox_tray-installation)\n* [:bookmark: Examples](#bookmark-examples)\n  * [Intermezzo: Almost Stochastic Order - a better significance test for Deep Neural Networks](#intermezzo-almost-stochastic-order---a-better-significance-test-for-deep-neural-networks)\n  * [Scenario 1: Comparing multiple runs of two models](#scenario-1---comparing-multiple-runs-of-two-models)\n  * [Scenario 2: Comparing multiple runs across datasets](#scenario-2---comparing-multiple-runs-across-datasets) \n  * [Scenario 3: Comparing sample-level scores](#scenario-3---comparing-sample-level-scores)\n  * [Scenario 4: Comparing more than two models](#scenario-4---comparing-more-than-two-models)\n  * [How to report results](#newspaper-how-to-report-results)\n  * [Sample size](#control_knobs-sample-size)\n  * [Other features](#sparkles-other-features)\n  * [General Recommendations \u0026 other notes](#general-recommendations) \n* [:mortar_board: Cite](#mortar_board-cite)\n* [:medal_sports: Acknowledgements](#medal_sports-acknowledgements)\n* [:people_holding_hands: Papers using deep-significance](#people_holding_hands-papers-using-deep-significance)\n* [:books: Bibliography](#books-bibliography)\n\n### :interrobang: Why?\n\nAlthough Deep Learning has undergone spectacular growth in the recent decade,\na large portion of experimental evidence is not supported by statistical hypothesis tests. Instead, \nconclusions are often drawn based on single performance scores. \n\nThis is problematic: Neural network display highly non-convex\nloss surfaces (Li et al., 2018) and their performance depends on the specific hyperparameters that were found, or stochastic factors \nlike Dropout masks, making comparisons between architectures more difficult. Based on comparing only (the mean of) a \nfew scores, **we often cannot \nconclude that one model type or algorithm is better than another**.\nThis endangers the progress in the field, as seeming success due to random chance might lead practitioners astray. \n\nFor instance, a recent study in Natural Language Processing by Narang et al. (2021) has found that many modifications proposed to \ntransformers do not actually improve performance. Similar issues are known to plague other fields like e.g., \nReinforcement Learning (Henderson et al., 2018) and Computer Vision (Borji, 2017) as well. \n\nTo help mitigate this problem, this package supplies fully-tested re-implementations of useful functions for significance\ntesting:\n* Statistical Significance tests such as Almost Stochastic Order (del Barrio et al, 2017; Dror et al., 2019), \n  bootstrap (Efron \u0026 Tibshirani, 1994) and permutation-randomization (Noreen, 1989).\n* Bonferroni correction methods for multiplicity in datasets (Bonferroni, 1936). \n* Bootstrap power analysis (Yuan \u0026 Hayashi, 2003) and other functions to determine the right sample size.\n\nAll functions are fully tested and also compatible with common deep learning data structures, such as PyTorch / \nTensorflow tensors as well as NumPy and Jax arrays.  For examples about the usage, consult the documentation \n[here](https://deep-significance.readthedocs.io/en/latest/) , the scenarios in the section [Examples](#examples) or \nthe [demo Jupyter notebook](https://github.com/Kaleidophon/deep-significance/tree/main/paper/deep-significance%20demo.ipynb).\n\n## :inbox_tray: Installation\n\nThe package can simply be installed using `pip` by running\n\n    pip3 install deepsig\n\nAnother option is to clone the repository and install the package locally:\n\n    git clone https://github.com/Kaleidophon/deep-significance.git\n    cd deep-significance\n    pip3 install -e .\n\n**Warning**: Installed like this, imports will fail when the clones repository is moved.\n\n## :bookmark: Examples\n\n---\n**tl;dr**: Use `aso()` to compare scores for two models. If the returned `eps_min \u003c 0.5`, A is better than B. The lower\n`eps_min`, the more confident the result (we recommend to check `eps_min \u003c 0.2` and record `eps_min` alongside \nexperimental results). \n\n:warning: Testing models with only one set of hyperparameters and only one test set will be able to guarantee superiority\nin all settings. See [General Recommendations \u0026 other notes](#general-recommendations).\n\n---\n\nIn the following, we will lay out three scenarios that describe common use cases for ML practitioners and how to apply \nthe methods implemented in this package accordingly. For an introduction into statistical hypothesis testing, please\nrefer to resources such as [this blog post](https://machinelearningmastery.com/statistical-hypothesis-tests/) for a general\noverview or [Dror et al. (2018)](https://www.aclweb.org/anthology/P18-1128.pdf) for a NLP-specific point of view. \n\nWe assume that we have two sets of scores we would like to compare, \u003cimg src=\"svgs/b7e817ab52abd984b082abaa1da6a8e4.svg?invert_in_darkmode\" align=middle width=17.44287434999999pt height=22.648391699999998pt/\u003e and \u003cimg src=\"svgs/d06f8d92c07734af06da289c13d2beed.svg?invert_in_darkmode\" align=middle width=16.80361814999999pt height=22.648391699999998pt/\u003e,\nfor instance obtained by running two models \u003cimg src=\"svgs/d41a53916d4850841d856bc8f5aa809a.svg?invert_in_darkmode\" align=middle width=11.87217899999999pt height=22.648391699999998pt/\u003e and \u003cimg src=\"svgs/f0e8ebc4201c3608138c518417f42ac4.svg?invert_in_darkmode\" align=middle width=10.95894029999999pt height=22.648391699999998pt/\u003e multiple times with a different random seed. \nWe can then define a one-sided test statistic  \u003cimg src=\"svgs/ae00ae93dc535f589522f8780b5aa275.svg?invert_in_darkmode\" align=middle width=63.909690899999994pt height=24.65753399999998pt/\u003e based on the gathered observations. \nAn example of such test statistics is for instance the difference in observation means. We then formulate the following null-hypothesis:\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"svgs/00160c684b3af8ccefcdf19c69712e34.svg?invert_in_darkmode\" align=middle width=128.7838134pt height=16.438356pt/\u003e\u003c/p\u003e\n\nThat means that we actually assume the opposite of our desired case, namely that \u003cimg src=\"svgs/d41a53916d4850841d856bc8f5aa809a.svg?invert_in_darkmode\" align=middle width=11.87217899999999pt height=22.648391699999998pt/\u003e is not better than \u003cimg src=\"svgs/f0e8ebc4201c3608138c518417f42ac4.svg?invert_in_darkmode\" align=middle width=10.95894029999999pt height=22.648391699999998pt/\u003e, \nbut equally as good or worse, as indicated by the value of the test statistic. \nUsually, the goal becomes to reject this null hypothesis using the SST. \n*p*-value testing is a frequentist method in the realm of SST. \nIt introduces the notion of data that *could have been observed* if we were to repeat our experiment again using \nthe same conditions, which we will write with superscript \u003cimg src=\"svgs/e723e08dae472a15132221e280670a7e.svg?invert_in_darkmode\" align=middle width=22.87678634999999pt height=14.15524440000002pt/\u003e in order to distinguish them from our actually \nobserved scores (Gelman et al., 2021). \nWe then define the *p*-value as the probability that, under the null hypothesis, the test statistic using replicated \nobservation is larger than or equal to the *observed* test statistic:\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"svgs/5db9dda6d48361ba963326d3f98a033d.svg?invert_in_darkmode\" align=middle width=216.90071865pt height=17.74869195pt/\u003e\u003c/p\u003e\n\nWe can interpret this expression as follows: Assuming that \u003cimg src=\"svgs/d41a53916d4850841d856bc8f5aa809a.svg?invert_in_darkmode\" align=middle width=11.87217899999999pt height=22.648391699999998pt/\u003e is not better than \u003cimg src=\"svgs/f0e8ebc4201c3608138c518417f42ac4.svg?invert_in_darkmode\" align=middle width=10.95894029999999pt height=22.648391699999998pt/\u003e, the test \nassumes a corresponding distribution of statistics that \u003cimg src=\"svgs/38f1e2a089e53d5c990a82f284948953.svg?invert_in_darkmode\" align=middle width=7.928075099999989pt height=22.831056599999986pt/\u003e is drawn from. So how does the observed test statistic \n\u003cimg src=\"svgs/ae00ae93dc535f589522f8780b5aa275.svg?invert_in_darkmode\" align=middle width=63.909690899999994pt height=24.65753399999998pt/\u003e fit in here? This is what the \u003cimg src=\"svgs/2ec6e630f199f589a2402fdf3e0289d5.svg?invert_in_darkmode\" align=middle width=8.270567249999992pt height=14.15524440000002pt/\u003e-value expresses: When the \nprobability is high, \u003cimg src=\"svgs/ae00ae93dc535f589522f8780b5aa275.svg?invert_in_darkmode\" align=middle width=63.909690899999994pt height=24.65753399999998pt/\u003e is in line with what we expected under the \nnull hypothesis, so we can *not* reject the null hypothesis, or in other words, we \\emph{cannot} conclude \n\u003cimg src=\"svgs/d41a53916d4850841d856bc8f5aa809a.svg?invert_in_darkmode\" align=middle width=11.87217899999999pt height=22.648391699999998pt/\u003e to be better than \u003cimg src=\"svgs/f0e8ebc4201c3608138c518417f42ac4.svg?invert_in_darkmode\" align=middle width=10.95894029999999pt height=22.648391699999998pt/\u003e. If the probability is low, that means that the observed \n\u003cimg src=\"svgs/67ebeedcf8c4d1141331d07b2cef2b03.svg?invert_in_darkmode\" align=middle width=54.77736824999999pt height=24.65753399999998pt/\u003e is quite unlikely under the null hypothesis and that the reverse case is \nmore likely - i.e. that it is likely larger than - and we conclude that \u003cimg src=\"svgs/d41a53916d4850841d856bc8f5aa809a.svg?invert_in_darkmode\" align=middle width=11.87217899999999pt height=22.648391699999998pt/\u003e is indeed better than \n\u003cimg src=\"svgs/f0e8ebc4201c3608138c518417f42ac4.svg?invert_in_darkmode\" align=middle width=10.95894029999999pt height=22.648391699999998pt/\u003e. Note that **the \u003cimg src=\"svgs/2ec6e630f199f589a2402fdf3e0289d5.svg?invert_in_darkmode\" align=middle width=8.270567249999992pt height=14.15524440000002pt/\u003e-value does not express whether the null hypothesis is true**. To make our decision \nabout whether or not to reject the null hypothesis, we typically determine a threshold - the significance level \n\u003cimg src=\"svgs/c745b9b57c145ec5577b82542b2df546.svg?invert_in_darkmode\" align=middle width=10.57650494999999pt height=14.15524440000002pt/\u003e, often set to 0.05 - that the *p*-value has to fall below. However, it has been argued that a better practice \ninvolves reporting the *p*-value alongside the results without a pidgeonholing of results into significant and non-significant\n(Wasserstein et al., 2019).\n\n\n### Intermezzo: Almost Stochastic Order - a better significance test for Deep Neural Networks\n\nDeep neural networks are highly non-linear models, having their performance highly dependent on hyperparameters, random \nseeds and other (stochastic) factors. Therefore, comparing the means of two models across several runs might not be \nenough to decide if a model A is better than B. In fact, **even aggregating more statistics like standard deviation, minimum\nor maximum might not be enough** to make a decision. For this reason, del Barrio et al. (2017) and Dror et al. (2019) \nintroduced *Almost Stochastic Order* (ASO), a test to compare two score distributions. \n\nIt builds on the concept of *stochastic order*: We can compare two distributions and declare one as *stochastically dominant*\nby comparing their cumulative distribution functions: \n\n![](img/so.png)\n\nHere, the CDF of A is given in red and in green for B. If the CDF of A is lower than B for every \u003cimg src=\"svgs/332cc365a4987aacce0ead01b8bdcc0b.svg?invert_in_darkmode\" align=middle width=9.39498779999999pt height=14.15524440000002pt/\u003e, we know the \nalgorithm A to score higher. However, in practice these cases are rarely so clear-cut (imagine e.g. two normal \ndistributions with the same mean but different variances).\nFor this reason, del Barrio et al. (2017) and Dror et al. (2019) consider the notion of *almost stochastic dominance* \nby quantifying the extent to which stochastic order is being violated (red area):\n\n![](img/aso.png)\n\nASO returns a value \u003cimg src=\"svgs/70bcb72c245ba47b6fc7439da91ec6fc.svg?invert_in_darkmode\" align=middle width=28.45332764999999pt height=14.15524440000002pt/\u003e, which expresses (an upper bound to) the amount of violation of stochastic order. If \n\u003cimg src=\"svgs/4cd4877610a47d915f39367760234822.svg?invert_in_darkmode\" align=middle width=60.239714699999986pt height=17.723762100000005pt/\u003e (where \\tau is 0.5 or less), A is stochastically dominant over B in more cases than vice versa, then the corresponding algorithm can be declared as \nsuperior. We can also interpret \u003cimg src=\"svgs/70bcb72c245ba47b6fc7439da91ec6fc.svg?invert_in_darkmode\" align=middle width=28.45332764999999pt height=14.15524440000002pt/\u003e as a *confidence score*. The lower it is, the more sure we can be \nthat A is better than B. Note: **ASO does not compute p-values.** Instead, the null hypothesis formulated as \n\n\u003cp align=\"center\"\u003e\u003cimg src=\"svgs/06f5ff6214110287d3948e9b44e31a1f.svg?invert_in_darkmode\" align=middle width=94.97699804999999pt height=13.698590399999999pt/\u003e\u003c/p\u003e\n\nIf we want to be more confident about the result of ASO, we can also set the rejection threshold to be lower than 0.5 \n(see the discussion in [this section](#general-recommendations)).\nFurthermore, the significance level \u003cimg src=\"svgs/c745b9b57c145ec5577b82542b2df546.svg?invert_in_darkmode\" align=middle width=10.57650494999999pt height=14.15524440000002pt/\u003e is determined as an input argument when running ASO and actively influence \nthe resulting \u003cimg src=\"svgs/70bcb72c245ba47b6fc7439da91ec6fc.svg?invert_in_darkmode\" align=middle width=28.45332764999999pt height=14.15524440000002pt/\u003e.\n\n\n### Scenario 1 - Comparing multiple runs of two models \n\nIn the simplest scenario, we have retrieved a set of scores from a model A and a baseline B on a dataset, stemming from \nvarious model runs with different seeds. We want to test whether our model A is better than B (higher scores = better)- \nWe can now simply apply the ASO test:\n\n```python\nimport numpy as np\nfrom deepsig import aso\n\nseed = 1234\nnp.random.seed(seed)\n\n# Simulate scores\nN = 5  # Number of random seeds\nmy_model_scores = np.random.normal(loc=0.9, scale=0.8, size=N)\nbaseline_scores = np.random.normal(loc=0, scale=1, size=N)\n\nmin_eps = aso(my_model_scores, baseline_scores, seed=seed)  # min_eps = 0.225, so A is better\n```\n\nNote that ASO **does not make any assumptions about the distributions of the scores**. \nThis means that we can apply it to any kind of test metric, as long as a higher score indicates a better performance \n(to apply ASO to cases where lower scores indicate better performances, just multiple your scores by -1 before feeding\nthem into the function). The more scores of model runs is supplied, the more reliable \nthe test becomes, so try to collect scores from as many runs as possible to reject the null hypothesis confidently.\n\n### Scenario 2 - Comparing multiple runs across datasets\n\nWhen comparing models across datasets, we formulate one null hypothesis per dataset. However, we have to make sure not to \nfall prey to the [multiple comparisons problem](https://en.wikipedia.org/wiki/Multiple_comparisons_problem): In short, \nthe more comparisons between A and B we are conducting, the more likely gets is to reject a null-hypothesis accidentally.\nThat is why we have to adjust our significance threshold \u003cimg src=\"svgs/c745b9b57c145ec5577b82542b2df546.svg?invert_in_darkmode\" align=middle width=10.57650494999999pt height=14.15524440000002pt/\u003e accordingly by dividing it by the number of comparisons, \nwhich corresponds to the Bonferroni correction (Bonferroni et al., 1936):\n\n```python \nimport numpy as np\nfrom deepsig import aso \n\nseed = 1234\nnp.random.seed(seed)\n\n# Simulate scores for three datasets\nM = 3  # Number of datasets\nN = 5  # Number of random seeds\nmy_model_scores_per_dataset = [np.random.normal(loc=0.3, scale=0.8, size=N) for _ in range(M)]\nbaseline_scores_per_dataset  = [np.random.normal(loc=0, scale=1, size=N) for _ in range(M)]\n\n# epsilon_min values with Bonferroni correction \neps_min = [aso(a, b, confidence_level=0.95, num_comparisons=M, seed=seed) for a, b in zip(my_model_scores_per_dataset, baseline_scores_per_dataset)]\n# eps_min = [0.006370113450148568, 0.6534772728574852, 0.0]\n```\n\n### Scenario 3 - Comparing sample-level scores\n\nIn previous examples, we have assumed that we compare two algorithms A and B based on their performance per run, i.e. \nwe run each algorithm once per random seed and obtain exactly one score on our test set. In some cases however, \nwe would like to compare two algorithms based on scores **for every point in the test set**. If we only use one seed\nper model, then this case is equivalent to scenario 1. But what if we also want to use multiple seeds per model?\n\nIn this scenario, we can do pair-wise comparisons of the score distributions between A and B and use the Bonferroni \ncorrection accordingly:\n\n```python \nfrom itertools import product \n\nimport numpy as np\nfrom deepsig import aso \n\nseed = 1234\nnp.random.seed(seed)\n\n# Simulate scores for three datasets\nM = 40   # Number of data points\nN = 3  # Number of random seeds\nmy_model_scored_samples_per_run = [np.random.normal(loc=0.3, scale=0.8, size=M) for _ in range(N)]\nbaseline_scored_samples_per_run = [np.random.normal(loc=0, scale=1, size=M) for _ in range(N)]\npairs = list(product(my_model_scored_samples_per_run, baseline_scored_samples_per_run))\n\n# epsilon_min values with Bonferroni correction \neps_min = [aso(a, b, confidence_level=0.95, num_comparisons=len(pairs), seed=seed) for a, b in pairs]\n# eps_min = [0.3831678636198528, 0.07194780234194881, 0.9152792807128325, 0.5273463008857844, 0.14946944524461184, 1.0, \n# 0.6099543280369378, 0.22387448804041898, 1.0]\n```\n\n### Scenario 4 - Comparing more than two models \n\nSimilarly, when comparing multiple models (now again on a per-seed basis), we can use a similar approach like in the \nprevious example. For instance, for three models, we can create a \u003cimg src=\"svgs/9f2b6b0a7f3d99fd3f396a1515926eb3.svg?invert_in_darkmode\" align=middle width=36.52961069999999pt height=21.18721440000001pt/\u003e matrix and fill the entries \nwith the corresponding \u003cimg src=\"svgs/70bcb72c245ba47b6fc7439da91ec6fc.svg?invert_in_darkmode\" align=middle width=28.45332764999999pt height=14.15524440000002pt/\u003e values.\n\nThe package implements the function `multi_aso()` exactly for this purpose. It has the same arguments as `aso()`, with \na few differences. First of all, the function takes a single `scores` argument, which can be a list of lists (of scores),\nor a nested NumPy array or Tensorflow / PyTorch / Jax tensor or dictionary (more about that later). \nLet's look at an example:\n\n```python \nimport numpy as np \nfrom deepsig import multi_aso \n\nseed = 1234\nnp.random.seed(seed)\n \nN = 5  # Number of random seeds\nM = 3  # Number of different models / algorithms\n\n# Simulate different model scores by sampling from normal distributions with increasing means\n# Here, we will sample from N(0.1, 0.8), N(0.15, 0.8), N(0.2, 0.8)\nmy_models_scores = np.array([np.random.normal(loc=loc, scale=0.8, size=N) for loc in np.arange(0.1, 0.1 + 0.05 * M, step=0.05)])\n\neps_min = multi_aso(my_models_scores, confidence_level=0.95, seed=seed)\n    \n# eps_min =\n# array([[1.       , 0.92621655, 1.        ],\n#       [1.        , 1.        , 1.        ],\n#       [0.82081635, 0.73048716, 1.        ]])\n```\n\nIn the example, `eps_min` is now a matrix, containing the \u003cimg src=\"svgs/70bcb72c245ba47b6fc7439da91ec6fc.svg?invert_in_darkmode\" align=middle width=28.45332764999999pt height=14.15524440000002pt/\u003e score between all pairs of models (for \nthe same model, it set to 1 by default). The matrix is always to be read as ASO(row, column).\n\nThe function applies the bonferroni correction for multiple comparisons by \ndefault, but this can be turned off by using `use_bonferroni=False`.\n\nLastly, when the `scores` argument is a dictionary and the function is called with `return_df=True`, the resulting matrix is \ngiven as a `pandas.DataFrame` for increased readability:\n\n```python \nimport numpy as np \nfrom deepsig import multi_aso \n\nseed = 1234\nnp.random.seed(seed)\n \nN = 5  # Number of random seeds\nM = 3  # Number of different models / algorithms\n\n# Same setup as above, but use a dict for scores\nmy_models_scores = {\n  f\"model {i+1}\": np.random.normal(loc=loc, scale=0.8, size=N) \n  for i, loc in enumerate(np.arange(0.1, 0.1 + 0.05 * M, step=0.05))\n}\n\n# my_model_scores = {\n#   \"model 1\": array([...]),\n#   \"model 2\": array([...]),\n#   ...\n# }\n\neps_min = multi_aso(my_models_scores, confidence_level=0.95, return_df=True, seed=seed)\n    \n# This is now a DataFrame!\n# eps_min =\n#          model 1   model 2  model 3\n# model 1  1.000000  0.926217      1.0\n# model 2  1.000000  1.000000      1.0\n# model 3  0.820816  0.730487      1.0\n\n```\n\n### :newspaper: How to report results\n\nWhen ASO used, two important details have to be reported, namely the confidence level \u003cimg src=\"svgs/c745b9b57c145ec5577b82542b2df546.svg?invert_in_darkmode\" align=middle width=10.57650494999999pt height=14.15524440000002pt/\u003e and the \u003cimg src=\"svgs/70bcb72c245ba47b6fc7439da91ec6fc.svg?invert_in_darkmode\" align=middle width=28.45332764999999pt height=14.15524440000002pt/\u003e\nscore. Below lists some example snippets reporting the results of scenarios 1 and 4:\n\n    Using ASO with a confidence level $\\alpha = 0.05$, we found the score distribution of algorithm A based on three \n    random seeds to be stochastically dominant over B ($\\epsilon_\\text{min} = 0$).\n\n    We compared all pairs of models based on five random seeds each using ASO with a confidence level of \n    $\\alpha = 0.05$ (before adjusting for all pair-wise comparisons using the Bonferroni correction). Almost stochastic \n    dominance ($\\epsilon_\\text{min} \u003c \\tau$ with $\\tau = 0.2$) is indicated in table X.\n\n### :control_knobs: Sample size\n\nIt can be hard to determine whether the currently collected set of scores is large enough to allow for reliable \nsignificance testing or whether more scores are required. For this reason, `deep-significance` also implements functions to aid the decision of whether to \ncollect more samples or not. \n\nFirst of all, it contains *Bootstrap power analysis* (Yuan \u0026 Hayashi, 2003): Given a set of scores, it gives all of them a uniform lift to \ncreate an artificial, second sample. Then, the analysis runs repeated analyses using bootstrapped versions of both \nsamples, comparing them with a significance test. Ideally, this should yield a significant result: If the difference \nbetween the re-sampled original and the lifted sample is non-significant, the original sample has too high of a variance. The \nanalyses then returns the *percentage of comparisons* that yielded significant results. If the number is too low, \nmore scores should be collected and added to the sample. \n\nThe result of the analysis is the *statistical power*: The \nhigher the power, the smaller the risk of falling prey to a Type II error - the probability of mistakenly accepting the \nnull hypothesis, when in fact it should actually be rejected. Usually, a power of ~ 0.8 is recommended (although that is\nsometimes hard to achieve in a machine learning setup).\n\nThe function can be used in the following way:\n\n```python\nimport numpy as np\nfrom deepsig import bootstrap_power_analysis\n\nscores = np.random.normal(loc=0, scale=20, size=5)  # Create too small of a sample with high variance\npower = bootstrap_power_analysis(scores, show_progress=False)  # 0.081, way too low\n\nscores2 = np.random.normal(loc=0, scale=20, size=50)  # Let's collect more samples\npower2 = bootstrap_power_analysis(scores2, show_progress=False)  # Better power with 0.2556\n```\n\nBy default, `bootstrap_power_analysis()` uses a one-sided Welch's t-test. However, this can be modified by passing \na function to the `significance_test` argument, which expects a function taking two sets of scores and returning a \np-value.\n\nSecondly, if the Almost Stochastic Order test (ASO) is being used, there is a second function available. ASO estimates\nthe violation ratio of two samples using bootstrapping. However, there is necessarily some uncertainty around that \nestimate, given that we only possess a finite number of samples. Using more samples decreases the uncertainty and makes the estimate tighter.\nThe degree to which collecting more samples increases the tightness can be computed using the following function:\n\n```python\nimport numpy as np\nfrom deepsig import aso_uncertainty_reduction\n\nscores1 = np.random.normal(loc=0, scale=0.3, size=5)  # First sample with five scores\nscores2 = np.random.normal(loc=0.2, scale=5, size=3)  # Second sample with three scores\n\nred1 = aso_uncertainty_reduction(m_old=len(scores1), n_old=len(scores2), m_new=5, n_new=5)  # 1.1547005383792515\nred2 = aso_uncertainty_reduction(m_old=len(scores1), n_old=len(scores2), m_new=7, n_new=3)  # 1.0583005244258363\n\n# Adding two runs to scores1 increases tightness of estimate by 1.15\n# But adding two runs to scores2 only increases tightness by 1.06! So spending two more runs on scores1 is better\n```\n\n### :sparkles: Other features\n\n#### :rocket: For the impatient: ASO with multi-threading\n\nWaiting for all the bootstrap iterations to finish can feel tedious, especially when doing many comparisons. Therefore, \nASO supports multithreading using `joblib`\nvia the `num_jobs` argument. \n\n```python\nfrom deepsig import aso\nimport numpy as np\nfrom timeit import timeit\n\na = np.random.normal(size=1000)\nb = np.random.normal(size=1000)\n\nprint(timeit(lambda: aso(a, b, num_jobs=1, show_progress=False), number=5))  # 616.2249192680001\nprint(timeit(lambda: aso(a, b, num_jobs=4, show_progress=False), number=5))  # 208.05637107000007\n```\n\nIf you want to select the maximum number of jobs possible on your device, you can set `num_jobs=-1`:\n\n```pythons\nprint(timeit(lambda: aso(a, b, num_jobs=-1, show_progress=False), number=5))  # 187.26257274800003\n```\n\n#### :electric_plug: Compatibility with PyTorch, Tensorflow, Jax \u0026 Numpy\n\nAll tests implemented in this package also can take PyTorch / Tensorflow tensors and Jax or NumPy arrays as arguments:\n\n```python\nfrom deepsig import aso \nimport torch\n\na = torch.randn(5, 1)\nb = torch.randn(5, 1)\n\naso(a, b)  # It just works!\n```\n\n#### :woman_farmer: Setting seeds for replicability\n\nIn order to ensure replicability, both `aso()` and `multi_aso()` supply as `seed` argument. This even works \nwhen multiple jobs are used!\n\n#### :game_die: Permutation and bootstrap test \n\nShould you be suspicious of ASO and want to revert to the good old faithful tests, this package also implements \nthe paired-bootstrap as well as the permutation randomization test. Note that as discussed in the next section, these \ntests have less statistical power than ASO. Furthermore, a function for the Bonferroni-correction using \np-values can also be found using `from deepsig import bonferroni_correction`.\n\n```python3\nimport numpy as np\nfrom deepsig import bootstrap_test, permutation_test\n\na = np.random.normal(loc=0.8, size=10)\nb = np.random.normal(size=10)\n\nprint(permutation_test(a, b))  # 0.16183816183816183\nprint(bootstrap_test(a, b))    # 0.103\n```\n\n\n### General recommendations \u0026 other notes\n\n* Naturally, the CDFs built from `scores_a` and `scores_b` can only be approximations of the true distributions. Therefore,\nas many scores as possible should be collected, especially if the variance between runs is high. If only one run is available,\n  comparing sample-wise score distributions like in scenario 3 can be an option, but comparing multiple runs will \n  **always** be preferable. Ideally, scores should be obtained even using different sets of hyperparameters per model.\n  Because this is usually infeasible in practice, Bouthilier et al. (2020) recommend to **vary all other sources of variation**\n  between runs to obtain the most trustworthy estimate of the \"true\" performance, such as data shuffling, weight initialization etc.\n\n* `num_bootstrap_iterations` can be reduced to increase the speed of `aso()`. However, this is not \nrecommended as the result of the test will also become less accurate. Technically, \u003cimg src=\"svgs/70bcb72c245ba47b6fc7439da91ec6fc.svg?invert_in_darkmode\" align=middle width=28.45332764999999pt height=14.15524440000002pt/\u003e is a upper bound\n  that becomes tighter with the number of samples and bootstrap iterations (del Barrio et al., 2017). Thus, increasing \n  the number of jobs with `num_jobs` instead is always preferred.\n  \n* While we could declare a model stochastically dominant with \u003cimg src=\"svgs/dabed7f05cf133d9eb92631d564a96a8.svg?invert_in_darkmode\" align=middle width=72.19750559999999pt height=21.18721440000001pt/\u003e, we found this to have a comparatively high\nType I error (false positives). Tests [in our paper](https://arxiv.org/pdf/2204.06815.pdf) have shown that a more useful threshold that trades of Type I and \n  Type II error between different scenarios might be \u003cimg src=\"svgs/9ac49cb370a5b09fca29068ea18eab63.svg?invert_in_darkmode\" align=middle width=51.969107849999986pt height=21.18721440000001pt/\u003e.\n  \n* Bootstrap and permutation-randomization are all non-parametric tests, i.e. they don't make any assumptions about \nthe distribution of our test metric. Nevertheless, they differ in their *statistical power*, which is defined as the probability\n  that the null hypothesis is being rejected given that there is a difference between A and B. In other words, the more powerful \n  a test, the less conservative it is and the more it is able to pick up on smaller difference between A and B. Therefore, \n  if the distribution is known or found out why normality tests (like e.g. Anderson-Darling or Shapiro-Wilk), something like \n  a parametric test like Student's or Welch's t-test is preferable to bootstrap or permutation-randomization. However, \n  because these test are in turn less applicable in a Deep Learning setting due to the reasons elaborated on in \n  [Why?](#interrobang-why), ASO is still a better choice.\n\n### :mortar_board: Cite\n\nUsing this package in general, please cite the following:\n\n    @inproceedings{ulmer2022deep,\n      title={deep-significance: Easy and Meaningful Signifcance Testing in the Age of Neural Networks},\n      author={Ulmer, Dennis and Hardmeier, Christian and Frellsen, Jes},\n      booktitle={ML Evaluation Standards Workshop at the Tenth International Conference on Learning Representations},\n      year={2022}\n    }\n\n\nIf you use the ASO test via `aso()` or `multi_aso, please cite the original works:\n\n    @inproceedings{dror2019deep,\n      author    = {Rotem Dror and\n                   Segev Shlomov and\n                   Roi Reichart},\n      editor    = {Anna Korhonen and\n                   David R. Traum and\n                   Llu{\\'{\\i}}s M{\\`{a}}rquez},\n      title     = {Deep Dominance - How to Properly Compare Deep Neural Models},\n      booktitle = {Proceedings of the 57th Conference of the Association for Computational\n                   Linguistics, {ACL} 2019, Florence, Italy, July 28- August 2, 2019,\n                   Volume 1: Long Papers},\n      pages     = {2773--2785},\n      publisher = {Association for Computational Linguistics},\n      year      = {2019},\n      url       = {https://doi.org/10.18653/v1/p19-1266},\n      doi       = {10.18653/v1/p19-1266},\n      timestamp = {Tue, 28 Jan 2020 10:27:52 +0100},\n    }\n\n    @incollection{del2018optimal,\n      title={An optimal transportation approach for assessing almost stochastic order},\n      author={Del Barrio, Eustasio and Cuesta-Albertos, Juan A and Matr{\\'a}n, Carlos},\n      booktitle={The Mathematics of the Uncertain},\n      pages={33--44},\n      year={2018},\n      publisher={Springer}\n    }\n\nFor instance, you can write\n\n    In order to compare models, we use the Almost Stochastic Order test \\citep{del2018optimal, dror2019deep} as \n    implemented by \\citet{ulmer2022deep}.\n\n### :medal_sports: Acknowledgements\n\nThis package was created out of discussions of the [NLPnorth group](https://nlpnorth.github.io/) at the IT University \nCopenhagen, whose members I want to thank for their feedback. The code in this repository is in multiple places based on\nseveral of [Rotem Dror's](https://rtmdrr.github.io/) repositories, namely \n[this](https://github.com/rtmdrr/replicability-analysis-NLP), [this](https://github.com/rtmdrr/testSignificanceNLP)\nand [this one](https://github.com/rtmdrr/DeepComparison). Thanks also go out to her personally for being available to \nanswer questions and provide feedback to the implementation and documentation of this package.\n\nThe commit message template used in this project can be found [here](https://github.com/Kaleidophon/commit-template-for-humans).\nThe inline latex equations were rendered using [readme2latex](https://github.com/leegao/readme2tex).\n\n### :people_holding_hands: Papers using deep-significance\n\nIn this last section of the readme, I would like to refer to works already using `deep-significance`. Open an issue or \npull request if you would like to see your work added here!\n\n* [\"From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding\" (van der Groot et al., 2021)](https://robvanderg.github.io/doc/naacl2021.pdf)\n* [\"Cartography Active Learning\" (Zhang \u0026 Plank, 2021)](https://arxiv.org/pdf/2109.04282.pdf)\n* [\"SkillSpan: Hard and Soft Skill Extraction from English Job Postings\" (Zhang et al., 2022a)](https://arxiv.org/pdf/2204.12811.pdf)\n* [\"What do you mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification\" (Bassignana \u0026 Plank, 2022)](https://arxiv.org/pdf/2204.13516.pdf)\n* [\"KOMPETENCER: Fine-grained Skill Classification in Danish Job Postings\nvia Distant Supervision and Transfer Learning\" (Zhang et al., 2022b)](https://arxiv.org/pdf/2205.01381.pdf)\n* [\"Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method \u0026 Data Scarcity\" (Ulmer et al., 2022)](https://arxiv.org/pdf/2210.15452.pdf)  \n* [\"DBGSL: Dynamic Brain Graph Structure Learning\" (Campbell et al., 2022)](https://arxiv.org/pdf/2209.13513.pdf)\n* [\"Cross-lingual sentiment analysis of official EU Slavic languages\" (Pandurang Thakkar, 2022)](https://repozitorij.ffzg.unizg.hr/islandora/object/ffzg:7344)\n* [\"An Empirical Study on Cross-X Transfer for Legal Judgment Prediction\" (Niklaus et al., 2022)](https://arxiv.org/pdf/2209.12325.pdf)\n* [\"Why don't you do it right? analysing annotators' disagreement in subjective tasks\" (Sandri et al., 2023)](https://aclanthology.org/2023.eacl-main.178.pdf)\n* [\"Not all layers are equally as important: Every Layer Counts BERT\" (Charpentier et al., 2023)](https://arxiv.org/pdf/2311.02265)\n* [\"Semantics for vision-and-language understanding\" (Falcon, 2023)](https://air.uniud.it/bitstream/11390/1252364/2/PhD_thesis-4.pdf)\n* [\"Rethinking Relation Classification with Graph Meaning Representations\" (Zhou et al., 2023)](https://www.researchgate.net/profile/Li-Zhou-117/publication/377638708_Rethinking_Relation_Classification_with_Graph_Meaning_Representations/links/661e8f5043f8df018d142953/Rethinking-Relation-Classification-with-Graph-Meaning-Representations.pdf)\n* [\"Improving Legal Rhetorical Role Labeling Through Additional Data and Efficient Exploitation of Transformer Models\" (Gomes de Lima, 2023)](https://repositorio.ufrn.br/bitstream/123456789/56540/1/Melhorandorotulamentopapel_Lima_2023.pdf)\n* [\"Integration and Analysis of Biological Data for Computational Drug Discovery\" (Güvelinir, 2024)](https://open.metu.edu.tr/bitstream/handle/11511/104453/Heval_Atas-Guvenilir_Tez.pdf)\n* [\"Influence of Channel Selection and Subject’s Age on the Performance of the Single Channel EEG-Based Automatic Sleep Staging Algorithms\" (Nazih et al., 2023)](https://www.mdpi.com/1424-8220/23/2/899)\n* [\"DH-FBK at SemEval-2023 Task 10: Multi-Task Learning with Classifier Ensemble Agreement for Sexism Detection\" (Leonardelli et al., 2023)](https://aclanthology.org/2023.semeval-1.261.pdf)\n* [\"VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning\" (Wang et al., 2023)](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10136716)\n* [\"Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech\" (Ilias et al., 2023)](https://www.sciencedirect.com/science/article/pii/S0950705123005841)\n* [\"Evaluating Recent Legal Rhetorical Role Labeling Approaches Supported by Transformer Encoders\" (Gomes de Lima et al., 2023)](https://link.springer.com/chapter/10.1007/978-3-031-45389-2_2)\n* [\"A Noisy Elephant in the Room: Is Your Out-of-Distribution Detector Robust to Label Noise?\" (Humblot-Renaux et al., 2024)](https://arxiv.org/pdf/2404.01775.pdf)\n* [\"Non-Fluent Synthetic Target-Language Data Improve Neural Machine Translation\" (Sánchez-Cartagena et al., 2024)](https://ieeexplore.ieee.org/iel7/34/4359286/10321682.pdf?casa_token=KENr2WEJU0kAAAAA:Ak204bHNgsKSZDBwH7HWh_kf3Z61j1upr4-qUG3z3uv-9S9deEYy3baYRjUbsSFkMS7owmhRSj5xHGU)\n* [\"Bootstrapping LLM-based Task-oriented Dialogue Agents via Self-talk\" (Ulmer et al., 2024a)](https://arxiv.org/pdf/2401.05033)\n* [\"Non-Exchangeable Conformal Language Generation with Nearest Neighbors\" (Ulmer et al., 2024b)](https://arxiv.org/pdf/2402.00707)\n* [\"Calibrating Large Language Models Using Their Generations Only\" (Ulmer at al., 2024c)](https://arxiv.org/pdf/2403.05973)\n* [\"More Room for Language: Investigating the Effect of Retrieval on Language Models\" (Samuel et al., 2024)](https://arxiv.org/pdf/2404.10939)\n* [\"On retrospective k-space subsampling schemes for deep MRI reconstruction\" (Yiasemis et al., 2024)](https://www.sciencedirect.com/science/article/pii/S0730725X23002199)\n* [\"Enhancing Turkish Coreference Resolution: Insights from deep learning, dropped pronouns, and multilingual transfer learning\" (Arslan et al., 2024)](https://www.sciencedirect.com/science/article/pii/S0885230824000640)\n* [\"A multivariable sensor-agnostic framework for spatio-temporal air quality forecasting based on Deep Learning\" (Prado-Rujas et al., 2024)](https://www.sciencedirect.com/science/article/pii/S0952197623014550)\n* [\"Putting Context in Context: the Impact of Discussion Structure on Text Classification\" (Penzo et al., 2024)](https://arxiv.org/pdf/2402.02975)\n* [\"Transferring Sentiment Cross-Lingually within and across Same-Family Languages\" (Thakar et al., 2024)](https://www.mdpi.com/2076-3417/14/13/5652)\n* [\"Exploring Generalizable Distillation for Efficient Medical Image Segmentation\" (Qi et al., 2024)](https://ieeexplore.ieee.org/iel7/6221020/6363502/10491241.pdf?casa_token=dtdkNy75kl4AAAAA:fBAOmdpJVs0RBA4fh0hLHbl81e3oZgTRSejbITGD06U9DqjYCsNUUuRtOEemGtrKBTcsN_ucEQL5U2U)\n* [\"Explainable epidemiological thematic features for event based disease surveillance\" (Menya et al., 2024)](https://www.sciencedirect.com/science/article/pii/S0957417424007607)\n* [\"Time Series Classification for Detecting Parkinson’s Disease from Wrist Motions\" (Donie et al., 2024)](https://arxiv.org/pdf/2304.11265)\n* [\"What Can You Do with a Sword? Gender Biases in Text Game Affordances\" (McGuire \u0026 Tomuro, 2024)](https://link.springer.com/chapter/10.1007/978-3-031-49065-1_48)\n* [\"Transformer-Encoders for Mathematical Answer Retrieval\" (Reusch, 2024)](https://tud.qucosa.de/landing-page/?tx_dlf[id]=https%3A%2F%2Ftud.qucosa.de%2Fapi%2Fqucosa%253A91575%2Fmets)\n\n\n### :books: Bibliography\n\nDel Barrio, Eustasio, Juan A. Cuesta-Albertos, and Carlos Matrán. \"An optimal transportation approach for assessing almost stochastic order.\" The Mathematics of the Uncertain. Springer, Cham, 2018. 33-44.\n\nBonferroni, Carlo. \"Teoria statistica delle classi e calcolo delle probabilita.\" Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze 8 (1936): 3-62.\n\nBorji, Ali. \"Negative results in computer vision: A perspective.\" Image and Vision Computing 69 (2018): 1-8.\n\nBouthillier, Xavier, et al. \"Accounting for variance in machine learning benchmarks.\" Proceedings of Machine Learning and Systems 3 (2021).\n\nDror, Rotem, et al. \"The hitchhiker’s guide to testing statistical significance in natural language processing.\" Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.\n\nDror, Rotem, Shlomov, Segev, and Reichart, Roi. \"Deep dominance-how to properly compare deep neural models.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.\n\nEfron, Bradley, and Robert J. Tibshirani. \"An introduction to the bootstrap.\" CRC press, 1994.\n\nAndrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, Donald B Rubin, John\nCarlin, Hal Stern, Donald Rubin, and David Dunson. Bayesian data analysis third edition, 2021.\n\nHenderson, Peter, et al. \"Deep reinforcement learning that matters.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n\nHao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein. \"Visualizing the Loss Landscape of Neural Nets.\" NeurIPS 2018: 6391-6401\n\nNarang, Sharan, et al. \"Do Transformer Modifications Transfer Across Implementations and Applications?.\" arXiv preprint arXiv:2102.11972 (2021).\n\nNoreen, Eric W. \"Computer intensive methods for hypothesis testing: An introduction.\" Wiley, New York (1989).\n\nRonald L Wasserstein, Allen L Schirm, and Nicole A Lazar. Moving to a world beyond “p\u003c 0.05”,\n2019\n\nYuan, Ke‐Hai, and Kentaro Hayashi. \"Bootstrap approach to inference and power analysis based on three test statistics for covariance structure models.\" British Journal of Mathematical and Statistical Psychology 56.1 (2003): 93-110.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaleidophon%2Fdeep-significance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkaleidophon%2Fdeep-significance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkaleidophon%2Fdeep-significance/lists"}