{"id":25167156,"url":"https://github.com/biocomputingup/cafa-evaluator","last_synced_at":"2025-04-30T22:02:26.283Z","repository":{"id":63369327,"uuid":"420024063","full_name":"BioComputingUP/CAFA-evaluator","owner":"BioComputingUP","description":"A Python Tool for Benchmarking Ontological Classification Methods","archived":false,"fork":false,"pushed_at":"2024-05-18T21:23:04.000Z","size":3724,"stargazers_count":36,"open_issues_count":1,"forks_count":15,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-01-11T17:55:43.154Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BioComputingUP.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-10-22T08:34:32.000Z","updated_at":"2024-11-22T10:49:07.000Z","dependencies_parsed_at":"2023-02-15T08:16:33.558Z","dependency_job_id":"4270a1e2-423b-484d-ba51-a7c380527358","html_url":"https://github.com/BioComputingUP/CAFA-evaluator","commit_stats":{"total_commits":62,"total_committers":3,"mean_commits":"20.666666666666668","dds":"0.048387096774193505","last_synced_commit":"739fd61d593b4f75dbcfbf41781b6cf1a6b44425"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BioComputingUP%2FCAFA-evaluator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BioComputingUP%2FCAFA-evaluator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BioComputingUP%2FCAFA-evaluator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BioComputingUP%2FCAFA-evaluator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BioComputingUP","download_url":"https://codeload.github.com/BioComputingUP/CAFA-evaluator/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237911261,"owners_count":19386040,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-09T06:19:45.411Z","updated_at":"2025-02-09T06:19:46.074Z","avatar_url":"https://github.com/BioComputingUP.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CAFA-evaluator\n\nCAFA-evaluator is a Python program designed to evaluate the performance of prediction methods on targets \nwith hierarchical concept dependencies.\nIt generalizes multi-label evaluation to modern ontologies where the prediction targets are drawn \nfrom a directed acyclic graph and achieves high efficiency by leveraging matrix computation and topological sorting.\n\nThe code replicates the \n[Critical Assessment of protein Function Annotation (CAFA)](https://www.biofunctionprediction.org/cafa/) benchmarking, \nwhich evaluates predictions of the consistent subgraphs in [Gene Ontology](http://geneontology.org/). \nThe package also contains a **Jupyter Notebook** to generate precision-recall and remaining uncertainty–misinformation curves.\nCAFA-evaluator implementation was inspired by the original Matlab code used in CAFA2 assessment and available at \n[https://github.com/yuxjiang/CAFA2](https://github.com/yuxjiang/CAFA2).\n\nVisit the [CAFA-evaluator Wiki](https://github.com/BioComputingUP/CAFA-evaluator/wiki) for more information about the algorithm.\n\n## Citation\n[CAFA-evaluator: A Python Tool for Benchmarking Ontological Classification Methods](https://doi.org/10.1093/bioadv/vbae043)  \n*D Piovesan, D Zago, P Joshi, MC De Paolis Kaluza, M Mehdiabadi, R Ramola, AM Monzon, W Reade, I Friedberg, P Radivojac, SCE Tosatto*  \n**Bioinformatics Advances (2024)** - DOI: [10.1093/bioadv/vbae043](https://doi.org/10.1093/bioadv/vbae043)\n\n## Installation\n\nYou can use the tool simply cloning this repo and running `__main__.py` Python script. \nwhich implements a command line interface.\nTo avoid import errors you need to add the source root folder \nto the `PYTHONPATH` environment variable. For example:\n\n    export PYTHONPATH=/path/to/CAFA-evaluator/src:$PYTHONPATH\n\nAlternatively, you can install the package with Pip. No need to export any variable\nin this case.\n\nFrom GitHub:\n\n    git clone https://github.com/BioComputingUP/CAFA-evaluator.git  \n    pip install .\n\nFrom PyPI:\n\n    pip install cafaeval\n\n## Usage\n\nThe program can be executing the command line interface or as a library.\nBoth the command line and the library accept the following positional arguments:\n\n* **Ontology file** in OBO format\n\n* **Prediction folder** contain prediction files. Files can be organized into sub-folders, \nsub-folders are processed recursively and the sub-folder name is used as prefix for the method name\n\n* **Ground truth file** containing targets and associated ontology terms\n\nExample input files are provided inside the `data/example` folder. If you have installed the software using pip,\nthe folder is located in the Python `site-packages` directory.\n\n### Command line\n\nWhen executed from the command line the script logs information about the calculation in the console (standard error) and\nwill create a folder named `results` containing the evaluation results. \nA different folder can be specified using the `-out_dir` option. \n\nThe following options are available:\nWhen installed with pip the script is available as `cafaeval` command.\n\n```bashcon\ncafaeval ontology_file prediction_folder ground_truth_file \n```\nIf you simply cloned the repository:\n```bashcon\npython3 /path/to/CAFA-evaluator/src/cafaeval/__main__.py ontology_file prediction_folder ground_truth_file\n```\n\n### Library\n\nThe `cafa_eval` function is the main entry point of the package. \nBelow is reported an example using the example files provided in the `data/example` folder.\n\n\n```pycon\n\u003e\u003e\u003e import cafaeval\n\u003e\u003e\u003e from cafaeval.evaluation import cafa_eval\n\u003e\u003e\u003e cafa_eval(\"IDPO_disorder_function.obo\", \"predictions\", \"ground_truth.tsv\")\n(                                       n        tp        fp        fn        pr  ...         f         s  pr_micro  rc_micro   f_micro\nfilename   ns                tau                                                  ...                                                  \npred_5.tsv disorder_function 0.01  168.0  2.928571  7.083333  0.130952  0.292532  ...  0.448380  7.084544  0.292509  0.957198  0.448087\n                             0.02  168.0  2.928571  7.083333  0.130952  0.292532  ...  0.448380  7.084544  0.292509  0.957198  0.448087\n                             0.03  168.0  2.928571  7.077381  0.130952  0.292695  ...  0.448571  7.078592  0.292683  0.957198  0.448292\n                             0.04  168.0  2.928571  7.077381  0.130952  0.292695  ...  0.448571  7.078592  0.292683  0.957198  0.448292\n                             0.05  168.0  2.928571  7.077381  0.130952  0.292695  ...  0.448571  7.078592  0.292683  0.957198  0.448292\n...                                  ...       ...       ...       ...       ...  ...       ...       ...       ...       ...       ...\npred_1.tsv disorder_function 0.41    1.0  0.005952  0.017857  3.053571  0.250000  ...  0.003937  3.053624  0.250000  0.001946  0.003861\n                             0.42    1.0  0.005952  0.017857  3.053571  0.250000  ...  0.003937  3.053624  0.250000  0.001946  0.003861\n                             0.43    1.0  0.005952  0.017857  3.053571  0.250000  ...  0.003937  3.053624  0.250000  0.001946  0.003861\n                             0.44    1.0  0.005952  0.017857  3.053571  0.250000  ...  0.003937  3.053624  0.250000  0.001946  0.003861\n                             0.45    1.0  0.005952  0.011905  3.053571  0.333333  ...  0.003945  3.053595  0.333333  0.001946  0.003868\n\n[352 rows x 14 columns], {'f':                                        n        tp        fp        fn        pr  ...         s  pr_micro  rc_micro   f_micro  cov_max\nfilename   ns                tau                                                  ...                                                 \npred_1.tsv disorder_function 0.04  166.0  1.744048  2.095238  1.315476  0.466566  ...  2.473964  0.454264  0.570039  0.505608      1.0\npred_2.tsv disorder_function 0.84  163.0  1.755952  1.845238  1.303571  0.504499  ...  2.259248  0.487603  0.573930  0.527256      1.0\npred_3.tsv disorder_function 0.89  168.0  2.113095  1.601190  0.946429  0.638889  ...  1.859983  0.568910  0.690661  0.623902      1.0\npred_4.tsv disorder_function 0.06  168.0  2.333333  0.666667  0.726190  0.777778  ...  0.985798  0.777778  0.762646  0.770138      1.0\npred_5.tsv disorder_function 0.38  167.0  2.345238  1.952381  0.714286  0.596671  ...  2.078941  0.545706  0.766537  0.637540      1.0\n\n[5 rows x 15 columns], 's':                                        n        tp        fp        fn        pr  ...         s  pr_micro  rc_micro   f_micro  cov_max\nfilename   ns                tau                                                  ...                                                 \npred_1.tsv disorder_function 0.06  102.0  1.047619  0.297619  2.011905  0.816667  ...  2.033799  0.778761  0.342412  0.475676      1.0\npred_2.tsv disorder_function 0.91  124.0  1.196429  0.839286  1.863095  0.638978  ...  2.043410  0.587719  0.391051  0.469626      1.0\npred_3.tsv disorder_function 0.89  168.0  2.113095  1.601190  0.946429  0.638889  ...  1.859983  0.568910  0.690661  0.623902      1.0\npred_4.tsv disorder_function 0.06  168.0  2.333333  0.666667  0.726190  0.777778  ...  0.985798  0.777778  0.762646  0.770138      1.0\npred_5.tsv disorder_function 0.42  156.0  1.642857  0.714286  1.416667  0.776221  ...  1.586552  0.696970  0.536965  0.606593      1.0\n\n[5 rows x 15 columns], 'f_micro':                                        n        tp        fp        fn        pr  ...         s  pr_micro  rc_micro   f_micro  cov_max\nfilename   ns                tau                                                  ...                                                 \npred_1.tsv disorder_function 0.04  166.0  1.744048  2.095238  1.315476  0.466566  ...  2.473964  0.454264  0.570039  0.505608      1.0\npred_2.tsv disorder_function 0.84  163.0  1.755952  1.845238  1.303571  0.504499  ...  2.259248  0.487603  0.573930  0.527256      1.0\npred_3.tsv disorder_function 0.89  168.0  2.113095  1.601190  0.946429  0.638889  ...  1.859983  0.568910  0.690661  0.623902      1.0\npred_4.tsv disorder_function 0.06  168.0  2.333333  0.666667  0.726190  0.777778  ...  0.985798  0.777778  0.762646  0.770138      1.0\npred_5.tsv disorder_function 0.38  167.0  2.345238  1.952381  0.714286  0.596671  ...  2.078941  0.545706  0.766537  0.637540      1.0\n\n[5 rows x 15 columns]})\n```\n\nThe output of `cafa_eval` is a tuple containing:\n* A pandas DataFrame with the evaluation results, one row per prediction file, namespace and threshold.\n* A dictionary with the best scores (max F-measure, max Weighted F-measure, min Semantic similarity). For each \nscore the dictionary contain a pandas DataFrame with one row per prediction file, namespace and threshold. \n\nThe `write_results` function generates the output files.\n\n```pycon\n\u003e\u003e\u003e import cafaeval\n\u003e\u003e\u003e from cafaeval.evaluation import cafa_eval, write_results\n\u003e\u003e\u003e res = cafa_eval(\"IDPO_disorder_function.obo\", \"predictions\", \"ground_truth.tsv\")\n\u003e\u003e\u003e write_results(*res)\n```\n\n\n## Input files\n**Prediction file** - Tab separated file with the target ID, term ID and score columns.\n\n~~~txt\nT_1\tIDPO:00501\t0.06\nT_1\tIDPO:00506\t0.05\nT_1\tIDPO:00507\t0.03\nT_2\tIDPO:00501\t0.04\nT_2\tIDPO:00506\t0.02\n...\n~~~\n\n**Ground truth file** - Tab separated file with the target ID and term ID. \nAdditional columns are discarded.\n~~~\nT_1\tIDPO:00024\nT_2\tIDPO:00506\nT_3\tIDPO:00502\nT_4\tIDPO:00025\n...\n~~~\n\n**Information accretion file (optional)** - If not provided, the weighted and S statistics are not generated.\nInformation accretion (IA) can be calculated as described in\n[Wyatt and Radivojac, Bioinformatics, 2013](https://pubmed.ncbi.nlm.nih.gov/23813009/).\n\n```\nIDPO:00024  6.32\nIDPO:00506  12.04\nIDPO:00502  1.34\nIDPO:00025  0.56\n...\n```\n\n## Output files\n\nOutput files are generated in the `results` folder. The same files are gerated by both\nthe command line and the `write_results` function.\n\n* `evaluation_all.tsv` corresponds to the first object returned by the `cafa_eval` function.\n* `evaluation_best_\u003c metric \u003e.tsv` corresponds to the second object returned by the `cafa_eval` function. \nA different file for each metric is created.\n\n**Note**: Weighted scores are generated only if the *Information Accretion* file is provided.\n\n\n## Optional parameters\n\n|   Argument  | Default value | Description                                                                                                                                                                                                                                                |\n|:-----------:|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n|   -out_dir  |   'results' | Output directory (tsv files + log). Either relative to current path or absolute                                                                                                                                                                            |\n|     -ia     |            | Information accretion file                                                                                                                                                                                                                                 |\n| -no_orphans |  False (flag) | Exclude orphans nodes (e.g. roots) from the calculation                                                                                                                                                                                                    |\n|    -norm    |     'cafa'  | Normalization strategy. `cafa` normalize precision by the number of predicted targets and recall by the number of targets in the ground truth. `pred` normalize by the number of  predicted targets. `gt` normalize by the number of ground truth proteins |\n|    -prop    |     'max'  | Ancestor propagation strategy. `max` propagate the max score of the traversed subgraph iteratively. `fill` propagate with max until a different score is found                                                                                             |\n|   -th_step  |      0.01  | Step size of prediction score thresholds to consider in the range [0, 1). A smaller step, means more calculation                                                                                                                                           |\n|  -max_terms |            | Number of terms for protein and namespace to consider in the evaluation. Parsing stops when the target limit for every namespace is reached. The score is not checked, meaning that terms are not sorted before the check, and the check is performed before propagation.                                                                                                                                                                                  |\n|   -threads  |       4    | Parallel threads. `0` means use all available CPU threads. Do not use multi thread if you are short in memory                                                                                                                                              |\n\n\n\n## Plotting\n\nPlots are generated by running all the cells in the `plot.ipynb` Jupyter Notebook.\nIn order to generate the figures you need to manually modify the first cell of the notebook which \ncontains a few parameters: \n* the path to the input file generated in the evaluation`evaluation_all.tsv`, see [Usage](#usage) above.\n* the output folder\n* the metric (F-score, weighted F-score, S measure).\n* an optional file including information about methods aliases and groups. If provided, the results\nare presented selecting only one method per group. The file should look like:\n```\nfilename\tgroup\tlabel\npred_1.tsv\tBioLab\tBioLab_model_1\npred_2.tsv\tBioLab\tBioLab_model_2\npred_3.tsv\tJohnLab\tJohnLab_model_1\n```\n\nThe notebook generates the following files:\n\n* `fig_\u003c metric \u003e_\u003c name_space \u003e.png` A figure for each namespace in the dataframe and the selected metric. \nThe notebook generates one metric at the time, you have to modify the input cell to generate the plots for a different metric\n\n* `fig_\u003c metric \u003e.tsv` A file with the data points for the metric curves. One curve for each method.\n```\ngroup\tlabel\tns\ttau\tcov\twrc\twpr\twf\nINGA\tINGA_1\tbiological_process\t0.010\t0.993\t0.557\t0.094\t0.160\nINGA\tINGA_1\tbiological_process\t0.020\t0.993\t0.555\t0.094\t0.161\nINGA\tINGA_1\tbiological_process\t0.030\t0.993\t0.552\t0.095\t0.162\nINGA\tINGA_1\tbiological_process\t0.040\t0.993\t0.551\t0.095\t0.163\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiocomputingup%2Fcafa-evaluator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbiocomputingup%2Fcafa-evaluator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbiocomputingup%2Fcafa-evaluator/lists"}