{"id":16190322,"url":"https://github.com/lgrz/pairwise-ttest","last_synced_at":"2025-10-03T20:46:21.120Z","repository":{"id":137808265,"uuid":"137822499","full_name":"lgrz/pairwise-ttest","owner":"lgrz","description":"Scripts to perform pairwise t-test on TREC run files","archived":false,"fork":false,"pushed_at":"2021-07-08T04:52:22.000Z","size":9,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-29T06:04:45.923Z","etag":null,"topics":["bonferroni","err","evaluation","information-retrieval","map","ndcg","rbp","statistics","trec","ttest"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lgrz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-19T00:55:13.000Z","updated_at":"2024-04-14T07:12:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"892a294a-1c09-4167-b029-efb44e805b14","html_url":"https://github.com/lgrz/pairwise-ttest","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgrz%2Fpairwise-ttest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgrz%2Fpairwise-ttest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgrz%2Fpairwise-ttest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgrz%2Fpairwise-ttest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lgrz","download_url":"https://codeload.github.com/lgrz/pairwise-ttest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232213530,"owners_count":18489550,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bonferroni","err","evaluation","information-retrieval","map","ndcg","rbp","statistics","trec","ttest"],"created_at":"2024-10-10T07:39:09.518Z","updated_at":"2025-10-03T20:46:16.066Z","avatar_url":"https://github.com/lgrz.png","language":"Shell","readme":"# Pairwise t-test\n\nScripts to perform pairwise t-test on TREC run files.\n\n### Requirements\n\n* R\n    * [reshape2][reshape2]\n* [gdeval.pl][gdeval] - note this is a fork that adds options `-k \u003ccutoff\u003e` and\n  `-j \u003cmax judgment\u003e` to the original [trec-web/trec-web-2013][trecweb]\n* [trec\\_eval][treceval]\n* [rbp\\_eval][rbpeval]\n\n[reshape2]: https://cran.r-project.org/web/packages/reshape2/index.html\n[gdeval]: https://github.com/lgrz/trec-web-2013\n[treceval]: https://trec.nist.gov/trec_eval\n[rbpeval]: https://people.eng.unimelb.edu.au/ammoffat/rbp_eval-0.2.tar.gz\n[trecweb]: https://github.com/trec-web/trec-web-2013\n\n### Usage\n\nThere are two bash scripts to run. First run `pairwise-eval.sh` to evaluate the\nTREC run files. Then run `pairwise-ttest.sh` to compute statistical\nsignificance.\n\nThe bash scripts assume that `rbp_eval`, `gdeval.pl` and `trec_eval` can be\nfound in your `PATH` environment.\n\nTo compute a pairwise t-test of all run files in the `runs` directory for\nNDCG@10 using `foo.qrels` (which contains the relevance judgments), run\nthe following:\n\n```\n./pairwise-eval.sh ndcg 10 foo.qrels runs/*.run\n./pairwise-ttest.sh runs/*.run.ndcg10 \u003e result.txt\ncat result.txt\n```\n\nThe `pairwise-eval.sh` script can compute ERR, NDCG, RBP and MAP. `gdeval.pl`\nis used for ERR and NDCG, `rbp_eval` for RBP, and `trec_eval` is used for MAP.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgrz%2Fpairwise-ttest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flgrz%2Fpairwise-ttest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgrz%2Fpairwise-ttest/lists"}