{"id":43111225,"url":"https://github.com/eugene-yang/tarexp","last_synced_at":"2026-01-31T18:23:54.785Z","repository":{"id":57473397,"uuid":"425630324","full_name":"eugene-yang/tarexp","owner":"eugene-yang","description":"An opensource TAR framework for experiments and applications","archived":false,"fork":false,"pushed_at":"2024-03-18T01:56:41.000Z","size":4170,"stargazers_count":17,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-29T19:26:32.670Z","etag":null,"topics":["active-learning","high-recall-retrieval","technology-assisted-review"],"latest_commit_sha":null,"homepage":"https://eugene.zone/tarexp","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eugene-yang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-07T22:24:12.000Z","updated_at":"2025-06-04T11:24:02.000Z","dependencies_parsed_at":"2022-09-22T18:51:37.440Z","dependency_job_id":null,"html_url":"https://github.com/eugene-yang/tarexp","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/eugene-yang/tarexp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-yang%2Ftarexp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-yang%2Ftarexp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-yang%2Ftarexp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-yang%2Ftarexp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eugene-yang","download_url":"https://codeload.github.com/eugene-yang/tarexp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eugene-yang%2Ftarexp/sbom","scorecard":{"id":384668,"data":{"date":"2025-08-11","repo":{"name":"github.com/eugene-yang/tarexp","commit":"d23724a73175cfacbaa95dbaf160a34bc17a1ac0"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.2,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/sphinx.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":0,"reason":"dangerous workflow patterns detected","details":["Warn: script injection with untrusted input 'github.head_ref': .github/workflows/sphinx.yml:45"],"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/sphinx.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/eugene-yang/tarexp/sphinx.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/sphinx.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/eugene-yang/tarexp/sphinx.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/sphinx.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/eugene-yang/tarexp/sphinx.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/sphinx.yml:60: update your workflow using https://app.stepsecurity.io/secureworkflow/eugene-yang/tarexp/sphinx.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/sphinx.yml:132: update your workflow using https://app.stepsecurity.io/secureworkflow/eugene-yang/tarexp/sphinx.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/sphinx.yml:41","Warn: pipCommand not pinned by hash: .github/workflows/sphinx.yml:42","Warn: pipCommand not pinned by hash: .github/workflows/sphinx.yml:43","Warn: pipCommand not pinned by hash: .github/workflows/sphinx.yml:63","Info:   0 out of   3 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned","Info:   0 out of   4 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"15 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2018-34 / GHSA-2fc2-6r4j-p65h","Warn: Project is vulnerable to: PYSEC-2021-856 / GHSA-5545-2q6w-2gh6","Warn: Project is vulnerable to: PYSEC-2019-108 / GHSA-9fq2-x9r6-wfmf","Warn: Project is vulnerable to: PYSEC-2018-33 / GHSA-cw6w-4rcx-xphc","Warn: Project is vulnerable to: PYSEC-2021-857 / GHSA-f7c7-j99h-c22f","Warn: Project is vulnerable to: GHSA-fpfv-jqm9-f5jm","Warn: Project is vulnerable to: PYSEC-2017-1 / GHSA-frgw-fgh6-9g52","Warn: Project is vulnerable to: PYSEC-2020-73","Warn: Project is vulnerable to: PYSEC-2020-107 / GHSA-jjw5-xxj6-pcv5","Warn: Project is vulnerable to: PYSEC-2024-110 / GHSA-jw8x-6495-233v","Warn: Project is vulnerable to: PYSEC-2020-108","Warn: Project is vulnerable to: PYSEC-2019-156 / GHSA-xp76-357g-9wqq","Warn: Project is vulnerable to: PYSEC-2023-102","Warn: Project is vulnerable to: PYSEC-2023-114","Warn: Project is vulnerable to: PYSEC-2017-74"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-18T16:19:12.213Z","repository_id":57473397,"created_at":"2025-08-18T16:19:12.213Z","updated_at":"2025-08-18T16:19:12.213Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28949289,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-31T14:26:55.697Z","status":"ssl_error","status_checked_at":"2026-01-31T14:26:52.545Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["active-learning","high-recall-retrieval","technology-assisted-review"],"created_at":"2026-01-31T18:23:54.004Z","updated_at":"2026-01-31T18:23:54.774Z","avatar_url":"https://github.com/eugene-yang.png","language":"Python","readme":"# TARexp: A Python Framework for Technology-Assisted Review Experiments\n\n`TARexp` is an opensource Python framework for conducting TAR experiments with various\nreference implementation to algorithms and methods that are commonly-used.\n\nThe experiments are fully reproducible and easy to conduct ablation studies. \nFor studying components that do not change the selection process of the review documents, \n`TARexp` supports replying TAR runs and experimenting these components offline. \n\nHelper functions to support results analysis are also avaiable. \n\nPlease visit our Google Colab Demo to check out the full running example [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eugene-yang/tarexp/blob/main/examples/exp-demo.ipynb)\n\nPlease refer to the documentation for more detail: https://eugene.zone/tarexp. \n\n## Get Started\n\nYou can install `TARexp` from PyPi by running\n```bash\npip install tarexp\n```\n\nOr install it with the lastest version from GitHub\n```bash\npip install git+https://github.com/eugene-yang/tarexp.git\n```\n\nIf you like to build it from source, please use\n```bash\ngit clone https://github.com/eugene-yang/tarexp.git\ncd tarexp\npython setup.py bdist_wheel\npip install dist/*.whl\n```\n\nIn Python, please use the following command to import both the main package and the components\n```python\nimport tarexp\nfrom tarexp import component\n```\n\n## Running Workflow\n\nThe following snippet is an example of creating a `dataset` instance for `TARexp`. \nFor `scikit-learn` rankers, the structure of the dataset is bascially a sparse `scipy` matrix\nfor the vectorized dataset and a list or an array of binary labels with the same length of the matrix. \n\n```python\nfrom sklearn import datasets\nimport pandas as pd\nrcv1 = datasets.fetch_rcv1()\nX = rcv1['data']\nrel_info = pd.DataFrame(rcv1['target'].todense().astype(bool), columns=rcv1['target_names'])\nds = tarexp.SparseVectorDataset.from_sparse(X)\n```\n\nThe following snippet defines a set of componets to use for a workflow, \n```python\nsetting = component.combine(component.SklearnRanker(LogisticRegression, solver='liblinear'), \n                            component.PerfectLabeler(), \n                            component.RelevanceSampler(), \n                            component.FixedRoundStoppingRule(max_round=20))()\n```\n\nAnd to declare a workflow, simply put in your dataset, setting, and other parameters to the workflow. \n```python\nworkflow = tarexp.OnePhaseTARWorkflow(\n    ds.set_label(rel_info['GPRO']), \n    setting, \n    seed_doc=[1023], \n    batch_size=200, \n    random_seed=123\n)\n```\n\nAnd finally, you can start executing the workflow by running it as an iterator. \nWe also support everything from [`ir-measures`](https://ir-measur.es/en/latest/) as evaluation metrics.\n\n```python\nrecording_metrics = [ir_measures.RPrec, tarexp.OptimisticCost(target_recall=0.8, cost_structure=(25,5,5,1))]\nfor ledger in workflow:\n    print(\"Round {}: found {} positives in total\".format(ledger.n_rounds, ledger.n_pos_annotated)) \n    print(\"metric:\", workflow.getMetrics(recording_metrics))\n```\n\nBesides standard IR evaluation metrics, we also implement `OptimisticCost` as cost-based evaluation metrics in `TARexp`. Please refer to [this paper](https://arxiv.org/abs/2106.09866) for more information and consider citing it if you use this measurement. \n\n## Running Experiments\n\n### TAR Experiments\n\n`tarexp.TARExperiment` is a wrapper and dispatcher for running TAR experiments with different settings. \nIt construct all combinations of the input settings and dispath each TAR run to execute.\n\nThe following command defines a set of 6 TAR runs which consists of 3 topics and each has 2 runs with batch size 200 and 100.  \n\n```python\nexp = tarexp.TARExperiment('./my_tar_exp/', random_seed=123, max_round_exec=20,\n                            metrics=[RPrec, P@10, tarexp.OptimisticCost(target_recall=0.8, cost_structure=(1,10,1,10))],\n                            tasks=tarexp.TaskFeeder(ds, rel_info[['GPRO', 'GOBIT', 'E141']]),\n                            components=setting,\n                            workflow=tarexp.OnePhaseTARWorkflow, batch_size=[200, 100])\n```\n\nTo start running the experiment, please use the following command which will execute with single processor and resume from any crash runs if exist in the output directory. \n```python\nresults = exp.run(n_processes=1, resume=True, dump_frequency=10)\n```\n\n### Testing Stopping Rules\n\n`TARexp` also encourages experiments on stopping rules. \nWe have built-in a number of stopping rules in the package and continuing to update them. \n\nThe following snippet is an exmaple for running a replay experiment based on a set of existing \nTAR runs with a list of stopping rules defined in `stopping_rules` arguments. \n\n```python\nreplay_exp = tarexp.StoppingExperimentOnReplay(\n                    './test_stopping_rules', random_seed=123,\n                    tasks=tarexp.TaskFeeder(ds, rel_info[['GPRO','GOBIT', 'E141']]),\n                    replay=tarexp.OnePhaseTARWorkflowReplay,\n                    saved_exp_path='./my_tar_exp',\n                    metrics=[tarexp.OptimisticCost(target_recall=0.8, cost_structure=(1,1,1,1)),\n                             tarexp.OptimisticCost(target_recall=0.9, cost_structure=(1,1,1,1))],\n                    stopping_rules=[\n                        component.KneeStoppingRule(), \n                        component.BudgetStoppingRule(), \n                        component.BatchPrecStoppingRule(), \n                        component.ReviewHalfStoppingRule(),\n                        component.Rule2399StoppingRule(), \n                        component.QuantStoppingRule(0.4, 0), \n                        component.QuantStoppingRule(0.2, 0),\n                        component.QuantStoppingRule(0.8, 0),\n                        component.CHMHeuristicsStoppingRule(0.8),\n                        component.CHMHeuristicsStoppingRule(0.4),\n                        component.CHMHeuristicsStoppingRule(0.2),\n                    ]\n            )\n\nstopping_results = replay_exp.run(resume=True, dump_frequency=10)\n```\n\n### Visualization\n\n`TARexp` also provide visualization tools for TAR runs. \n\n`createDFfromResults` creates a pandas DataFrame from either the result variable\n```python\ndf = tarexp.helper.createDFfromResults(results, remove_redundant_level=True)\n```\nOr the output directory\n```python\ndf = tarexp.helper.createDFfromResults('./my_tar_exp', remove_redundant_level=True)\n```\n\nAnd the following command provides you the cost dynamic graph introduced in [this paper](https://arxiv.org/abs/2106.09866). \n```python\ntarexp.helper.cost_dynamic(\n    df.loc[:, 'GOBIT', :].groupby(level='dataset'),\n    recall_targets=[0.8], cost_structures=[(1,1,1,1), (10, 10, 1, 1), (25, 5, 5, 1)],\n    with_hatches=True\n)\n```\n\n![](./examples/cost-dynamic-1.png)\n\nAlternatively, you can also create this graph by using a command line interface\n```bash\npython -m tarexp.helper.plotting \\\n       --runs GPRO=./my_tar_exp/GPRO.61b1f31a0a29de634939db77c0dde383/  \\\n              GOBIT=./my_tar_exp/GOBIT.ae86e0b37809cb139dfa1f4cf914fb9b/  \\\n       --cost_structures 1-1-1-1 25-5-5-1 --y_thousands --with_hatches\n```\n\n![](./examples/cost-dynamic-2.png)\n\n## Feedback\n\nAny feedback is welcome! \nYou can reach out to us either by emailing the author or rasing an issue! \n\n## Reference\n\nThe demo paper of `TARexp` is currently under review. \n\nIf you use the cost measure or the cost dynamic graphs, pleas consider citing this paper\n```bibtex\n@inproceedings{cost-structure,\n\tauthor = {Eugene Yang and David D. Lewis and Ophir Frieder},\n\ttitle = {On Minimizing Cost in Legal Document Review Workflows},\n\tbooktitle = {Proceedings of the ACM Symposium on Document Engineering (DocEng)},\n\tyear = {2021},\n\turl = {https://arxiv.org/abs/2106.09866}\n}\n```\n\n\n\n\n\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feugene-yang%2Ftarexp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feugene-yang%2Ftarexp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feugene-yang%2Ftarexp/lists"}