{"id":22731040,"url":"https://github.com/genentech/molcap-arena","last_synced_at":"2025-04-12T17:52:33.108Z","repository":{"id":260965896,"uuid":"881134669","full_name":"Genentech/molcap-arena","owner":"Genentech","description":"Associated Repository for \"MolCap-Arena: A Comprehensive Captioning Benchmark on Language-Enhanced Molecular Property Prediction\"","archived":false,"fork":false,"pushed_at":"2024-11-04T03:17:52.000Z","size":1360,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-26T12:12:00.553Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Genentech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-31T01:01:40.000Z","updated_at":"2025-02-18T03:29:52.000Z","dependencies_parsed_at":"2024-11-03T23:23:06.622Z","dependency_job_id":"9dc146f3-66ad-4eaf-b272-5eb59d884e01","html_url":"https://github.com/Genentech/molcap-arena","commit_stats":null,"previous_names":["genentech/molcap-arena"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genentech%2Fmolcap-arena","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genentech%2Fmolcap-arena/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genentech%2Fmolcap-arena/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genentech%2Fmolcap-arena/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Genentech","download_url":"https://codeload.github.com/Genentech/molcap-arena/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248610407,"owners_count":21132920,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-10T19:19:13.110Z","updated_at":"2025-04-12T17:52:33.087Z","avatar_url":"https://github.com/Genentech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MolCap-Arena: A Comprehensive Captioning Benchmark on Language-Enhanced Molecular Property Prediction\nAssociated repository for \"[MolCap-Arena: A Comprehensive Captioning Benchmark on Language-Enhanced Molecular Property Prediction](https://arxiv.org/abs/2411.00737)\".\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"figure.png\" width=\"70%\"\u003e\n\u003c/p\u003e\n\n## Table of Contents\n - [Submission](#submission)\n - [Installation](#installation)\n - [Running the benchmark](#running-the-benchmark)\n - [Evaluating a new model](#evaluating-a-new-model)\n - [Creating a new captioner](#creating-a-new-captioner)\n - [Adding a new dataset](#adding-a-new-dataset)\n - [Downloads](#downloads)\n - [Citation](#citation)\n - [Leaderboard](#leaderboard)\n\n\n### Submission \n\n(link coming soon)\n\nIf you would like to submit a caption file for easy distribution for others to use when building a leaderboard, please submit [here](). It should follow the format of existing files and contain captions for molecules in all datasets.  \n\n### Installation\n\nThere are two conda environments: 'MolCapArena' for evaluation, and 'Captioner' for captioning. \n\n```\nconda env create -f environment.yml\n\nconda env create -f environment_captioner.yml\n\nconda activate MolCapArena\npip install -e .\n```\n\n\n### Running the benchmark\n\nDownload the 'battles.zip' file from [Downloads](#downloads). \n\n```\nunzip battles.zip\nconda activate MolCapArena\npython leaderboard_scripts/create_metrics_h2h.py\npython leaderboard_scripts/create_winrate_plot_h2h.py\n```\n\n\n\n### Evaluating a new model\n\nCreate a new file in captions called 'YOUR_NAME.csv'. It should contain a header line with at least two fields: 'SMILES' and 'captions'. All canonicalized SMILES from 'all_smiles.csv' should appear. \n\nDownload the 'battles.zip', 'captions.zip', 'embeddings.zip', 'splits.zip' files from [Downloads](#downloads). \n\n```\nunzip battles.zip\nunzip captions.zip\nunzip embeddings.zip\nunzip splits.zip\n```\n\nSet important variables:\n```\ndeclare -a datasets=('BBBP' 'BACE' 'ClinTox' 'FreeSolv' 'ESOL' 'Lipo')\nexport DATASETS=\"${datasets[@]}\"\nexport caption_name='YOUR_NAME'\nexport WANDB_MODE=disabled\nconda activate MolCapArena\n```\n\nRun the four stages of the evaluation procedure: 1) model training, 2) embedding extraction, 3) multimodal SVM training, and 4) head-to-head captioner vs captioner battles. \n```\necho \"Training caption embedding models\"\nfor dataset in \"${datasets[@]}\"; do\n    for run in {0..4}; do\n        echo \"$dataset:$run\"\n        python leaderboard_scripts/run_train.py --specific_caption=\"$caption_name\" --dataset=\"$dataset\" --max_epochs=20 --validate_every_n=0 --fold_idx=$run\n    done\ndone\n\n\necho \"Extracting captions\"\nfor dataset in \"${datasets[@]}\"; do\n    for run in {0..4}; do\n        echo \"$dataset:$run\"\n        python leaderboard_scripts/run_embeddings.py --specific_caption=\"$caption_name\" --dataset=$dataset --fold_idx=$run --fold_idx=$run\n    done\ndone\n\necho \"Training single-caption predictors\"\nfor dataset in \"${datasets[@]}\"; do\n    for run in {0..4}; do\n        echo \"$dataset:$run\"\n        python leaderboard_scripts/head.py --dataset=$dataset --method=SVM --fold_idx=$run\n    done\ndone\n\necho \"Running head-to-head battles\"\nfor dataset in \"${datasets[@]}\"; do\n    for run in {0..4}; do\n        echo \"$dataset:$run\"\n        python leaderboard_scripts/head2head.py --dataset=$dataset --method=SVM --fold_idx=$run\n    done\ndone\n```\n\nFinally, evaluate the leaderboard:\n```\npython leaderboard_scripts/create_metrics_h2h.py\n```\n\n\n\n### Creating a new captioner\n\nCreate a new captioner file by first copying an existing file as an example. Modify 'get_all_captions.py' to include YOUR_NAME as an option. \n\n```\nconda activate Captioner\n\ncp captioners/Llama3_8B_captioner.py captioners/YOUR_NAME_captioner.py\n\necho \"Modify 'YOUR_NAME_captioner.py' and add an if statement to 'get_all_captions.py' to include it as an option.\"\n\npython get_all_captions.py --model=YOUR_NAME --debug\n\npython get_all_captions.py --model=YOUR_NAME\n\n```\n\n### Adding a new dataset\n\nCurrently, adding a new dataset is not fully supported. The rough steps are as follows:\n\n1. Download and unzip 'splits.zip' files from [Downloads](#downloads). \n2. Generate a csv file for the dataset in 'splits/'. Modify 'create_splits.py' to include your dataset, then run it.\n3. Modify 'build_smiles_list.py' to include the new dataset. Run 'build_smiles_list.py' then 'create_BRICS.py'.\n4. Run captioners to generate new caption files.\n    4.1. (Optional) Modify 'get_all_captions.py' to avoid re-captioning old molecules.\n5. Run leaderboard_scripts as detailed above. Note that scripts require modification to accomodate new datasets. \n\n\n### Downloads\n\n+ [battles](https://drive.google.com/file/d/1ohKJ2ISMnYvHYdLBPu-cZ32AX19xFiAa/view?usp=sharing)\n+ [captions](https://drive.google.com/file/d/1R84VBuk6o5VlAymLmMjEjchNStXK9yTG/view?usp=sharing)\n+ [embeddings](https://drive.google.com/file/d/1DbhEkawsplArg6VFmkAQbjuuRiemjnSq/view?usp=sharing)\n+ [splits](https://drive.google.com/file/d/1ep2-mg3G2kMKKEF3Bmd69OUuf2D8psvr/view?usp=sharing)\n\n\n### Citation\nIf you found our work useful, please cite:\n```bibtex\n@article{edwards2024molcaparena,\n  title={MolCap-Arena: A Comprehensive Captioning Benchmark on Language-Enhanced Molecular Property Prediction},\n  author={Edwards, Carl and Lu, Ziqing and Hajiramezanali, Ehsan and\nBiancalani, Tommaso and Ji, Heng and Scalia, Gabriele},\n  journal={arXiv preprint arXiv:2411.00737},\n  url={https://arxiv.org/abs/2411.00737}\n  year={2024}\n}\n```\n\n\n### Leaderboard\n\nHere is the leaderboard conveniently generated by GPT-4o (so results might be hallucinated). Please see [leaderboard_table.csv](leaderboard_table.csv) for real results. \n\n\n| Model                           | Rating  | 95% CI | ROC-AUC | BCE Loss | Avg. Error | Average Precision | Pearson R | Spearman R | R2    | MSE   | MAE  |\n|---------------------------------|---------|--------|---------|----------|-------------|--------------------|-----------|------------|-------|-------|------|\n| BioT5_plus                      | **1061**    | +3/-2| 87.359  | 0.335    | **0.471**       | 74.907             | 0.654     | 0.660      | 0.408 | 1.057 | 0.744|\n| BioT5                           | 1059    | +3/-2| **87.371**  | **0.334**    | 0.472       | 74.631             | 0.654     | 0.658      | 0.409 | 1.057 | 0.744|\n| LlaSMol                         | 1040    | +2/-3| 86.522  | 0.340    | 0.473       | 73.271             | 0.655     | 0.661      | 0.409 | 1.054 | 0.746|\n| GPT-4o-Generic                  | 1032    | +2/-2| 85.685  | 0.341    | 0.472       | 72.388             | 0.657     | 0.662      | 0.412 | 1.049 | 0.742|\n| TextChemT5                      | 1032    | +4/-2| 87.074  | 0.341    | 0.475       | **75.295**             | 0.652     | 0.657      | 0.407 | 1.057 | 0.746|\n| Gemma2-9B-Frags-Drug            | 1031    | +3/-4| 85.543  | 0.343    | 0.475       | 72.435             | 0.653     | 0.657      | 0.407 | 1.062 | 0.747|\n| Gemma2-9B-Frags-Chem            | 1030    | +3/-4| 85.619  | 0.342    | 0.475       | 72.312             | 0.652     | 0.657      | 0.406 | 1.061 | 0.747|\n| Llama3.1-70B-Chem               | 1021    | +2/-2| 85.431  | 0.344    | 0.475       | 72.529             | 0.653     | 0.656      | 0.408 | 1.056 | 0.746|\n| Llama3.1-70B-Generic            | 1021    | +1/-3| 85.618  | 0.344    | 0.474       | 72.373             | 0.653     | 0.657      | 0.408 | 1.054 | 0.744|\n| MolT5                           | 1021    | +1/-4| 86.406  | 0.340    | 0.475       | 75.251             | 0.652     | 0.655      | 0.406 | 1.064 | 0.748|\n| Gemma2-9B-Frags-Generic         | 1020    | +3/-3| 85.550  | 0.342    | 0.476       | 72.455             | 0.651     | 0.657      | 0.406 | 1.064 | 0.749|\n| Llama3.1-70B-Drug               | 1020    | +1/-2| 85.594  | 0.343    | **0.471**       | 72.430             | 0.655     | 0.660      | 0.412 | 1.042 | 0.740|\n| Llama3.1-405B-Generic           | 1020    | +3/-1| 85.469  | 0.342    | **0.471**       | 72.048             | **0.658**     | **0.663**      | **0.416** | **1.038** | **0.738**|\n| Llama3-70B-Drug                 | 1017    | +2/-3| 85.392  | 0.343    | 0.472       | 72.179             | 0.656     | 0.660      | 0.413 | 1.047 | 0.741|\n| MistralNeMo-12B-Frags-Generic   | 1017    | +2/-1| 85.376  | 0.345    | 0.473       | 72.022             | 0.653     | 0.658      | 0.408 | 1.051 | 0.742|\n| Llama3.1-70B-Bio                | 1016    | +2/-2| 85.238  | 0.347    | 0.473       | 72.074             | 0.656     | 0.661      | 0.411 | 1.047 | 0.741|\n| Gemma2-27B-Generic              | 1015    | +3/-3| 85.254  | 0.343    | **0.471**       | 72.152             | 0.654     | 0.661      | 0.412 | 1.043 | 0.739|\n| Llama3-70B-Chem                 | 1014    | +2/-4| 85.355  | 0.344    | 0.473       | 72.181             | 0.655     | 0.659      | 0.409 | 1.053 | 0.742|\n| Gemma2-27B-Chem                 | 1013    | +1/-3| 85.223  | 0.345    | 0.474       | 72.057             | 0.653     | 0.659      | 0.409 | 1.051 | 0.743|\n| MistralNeMo-12B-Frags-Chem      | 1012    | +2/-3| 85.746  | 0.343    | 0.472       | 72.320             | 0.656     | 0.660      | 0.411 | 1.046 | 0.741|\n| GPT-4o-Frags-Generic            | 1012    | +3/-2| 85.510  | 0.343    | 0.475       | 72.210             | 0.654     | 0.658      | 0.408 | 1.054 | 0.745|\n| Llama3-8B-Task                  | 1009    | +1/-2| 85.414  | 0.344    | 0.477       | 72.182             | 0.652     | 0.656      | 0.406 | 1.063 | 0.748|\n| MistralNeMo-12B-Frags-Drug      | 1008    | +3/-2| 85.731  | 0.344    | 0.473       | 72.436             | 0.654     | 0.659      | 0.410 | 1.046 | 0.742|\n| Llama3-70B-Generic              | 1008    | +3/-2| 85.108  | 0.345    | 0.475       | 71.920             | 0.653     | 0.656      | 0.406 | 1.061 | 0.746|\n| Gemma2-9B-Frags-Bio             | 1006    | +3/-3| 85.284  | 0.346    | 0.477       | 72.020             | 0.651     | 0.657      | 0.405 | 1.065 | 0.750|\n| Llama3.1-70B-Quant              | 1006    | +3/-3| 85.312  | 0.344    | 0.474       | 72.163             | 0.653     | 0.660      | 0.408 | 1.052 | 0.743|\n| Llama3.1-405B-Frags-Generic     | 1005    | +2/-2| 85.482  | 0.344    | 0.476       | 72.682             | 0.653     | 0.657      | 0.405 | 1.065 | 0.749|\n| Gemma2-9B-Chem                  | 1004    | +1/-2| 85.455  | 0.343    | 0.474       | 72.112             | 0.653     | 0.657      | 0.409 | 1.057 | 0.745|\n| Llama3-70B-Bio                  | 1003    | +1/-2| 85.479  | 0.344    | 0.476       | 72.253             | 0.651     | 0.656      | 0.403 | 1.066 | 0.748|\n| Llama3-70B-Frags-Generic        | 1003    | +3/-3| 85.780  | 0.342    | 0.478       | 72.693             | 0.650     | 0.655      | 0.403 | 1.072 | 0.751|\n| Llama3.1-8B-Drug                | 1003    | +4/-2| 85.088  | 0.345    | 0.475       | 72.004             | 0.654     | 0.657      | 0.410 | 1.052 | 0.745|\n| Llama3.1-70B-Frags-Generic      | 1001    | +3/-2| 85.580  | 0.342    | 0.477       | 72.238             | 0.650     | 0.656      | 0.403 | 1.071 | 0.750|\n| Llama3.1-70B-Frags-Chem         | 1001    | +3/-2| 85.512  | 0.343    | 0.477       | 72.306             | 0.651     | 0.657      | 0.404 | 1.070 | 0.749|\n| Gemma2-27B-Frags-Drug           | 1001    | +2/-4| 85.644  | 0.342    | 0.477       | 72.333             | 0.649     | 0.653      | 0.402 | 1.069 | 0.752|\n| MistralNeMo-12B-Bio             | 1000    | +1/-2| 85.301  | 0.344    | 0.476       | 72.144             | 0.650     | 0.655      | 0.403 | 1.066 | 0.748|\n| Llama3-8B-Drug                  | 999     | +3/-2| 85.375  | 0.344    | 0.474       | 72.148             | 0.654     | 0.658      | 0.408 | 1.057 | 0.745|\n| MistralNeMo-12B-Generic         | 999     | +1/-1| 85.421  | 0.343    | 0.477       | 72.113             | 0.650     | 0.654      | 0.403 | 1.069 | 0.750|\n| Gemma2-27B-Bio                  | 998     | +2/-1| 85.347  | 0.343    | 0.476       | 72.176             | 0.652     | 0.657      | 0.405 | 1.061 | 0.747|\n| Gemma2-9B-Bio                   | 997     | +3/-2| 85.392  | 0.344    | 0.475       | 72.139             | 0.651     | 0.656      | 0.406 | 1.057 | 0.746|\n| Gemma2-9B-Frags-Quant           | 997     | +5/-2| 85.305  | 0.346    | 0.475       | 72.035             | 0.652     | 0.656      | 0.406 | 1.060 | 0.746|\n| Gemma2-27B-Quant                | 997     | +3/-4| 85.348  | 0.346    | 0.476       | 72.120             | 0.651     | 0.656      | 0.405 | 1.061 | 0.746|\n| MistralNeMo-12B-Frags-Bio       | 996     | +2/-3| 85.364  | 0.345    | 0.475       | 72.082             | 0.652     | 0.658      | 0.406 | 1.060 | 0.746|\n| ChemDFM                         | 996     | +4/-3| 86.052  | 0.344    | 0.477       | 72.614             | 0.648     | 0.653      | 0.401 | 1.073 | 0.751|\n| MistralNeMo-12B-Chem            | 996     | +1/-5| 85.418  | 0.346    | 0.475       | 72.201             | 0.651     | 0.655      | 0.406 | 1.063 | 0.746|\n| 3D-MoLM                         | 996     | +2/-3| 86.032  | 0.341    | 0.475       | 73.161             | 0.651     | 0.655      | 0.406 | 1.062 | 0.748|\n| ChemLLM                         | 996     | +3/-3| 85.810  | 0.342    | 0.475       | 72.849             | 0.650     | 0.654      | 0.405 | 1.063 | 0.747|\n| Llama3-8B-Frags-Bio             | 996     | +2/-1| 85.338  | 0.344    | 0.477       | 72.102             | 0.652     | 0.657      | 0.405 | 1.071 | 0.750|\n| Llama3-8B-Quant                 | 995     | +3/-4| 85.397  | 0.344    | 0.476       | 72.154             | 0.650     | 0.654      | 0.405 | 1.064 | 0.747|\n| Llama3-8B-Frags-Chem            | 995     | +3/-2| 85.336  | 0.344    | 0.476       | 72.086             | 0.654     | 0.659      | 0.406 | 1.065 | 0.749|\n| Llama3.1-8B-Generic             | 995     | +2/-3| 85.331  | 0.343    | 0.476       | 72.231             | 0.652     | 0.657      | 0.406 | 1.062 | 0.748|\n| MistralNeMo-12B-Frags-Quant     | 994     | +3/-2| 85.194  | 0.344    | 0.476       | 72.143             | 0.651     | 0.655      | 0.403 | 1.065 | 0.749|\n| Llama3-70B-Frags-Drug           | 994     | +3/-2| 85.590  | 0.342    | 0.478       | 72.270             | 0.650     | 0.655      | 0.402 | 1.073 | 0.751|\n| Gemma2-27B-Drug                 | 994     | +3/-3| 85.332  | 0.344    | 0.473       | 72.067             | 0.653     | 0.659      | 0.409 | 1.045 | 0.741|\n| Llama3-70B-Quant                | 993     | +2/-2| 85.241  | 0.344    | 0.474       | 72.109             | 0.654     | 0.657      | 0.408 | 1.055 | 0.745|\n| Llama3-8B-Frags-Drug            | 993     | +2/-2| 85.352  | 0.343    | 0.477       | 72.144             | 0.652     | 0.656      | 0.406 | 1.064 | 0.748|\n| Gemma2-9B-Generic               | 993     | +4/-3| 85.412  | 0.344    | 0.476       | 72.085             | 0.650     | 0.654      | 0.404 | 1.067 | 0.748|\n| Llama3.1-8B-Chem                | 992     | +3/-2| 85.425  | 0.344    | 0.475       | 72.214             | 0.652     | 0.658      | 0.406 | 1.059 | 0.746|\n| MolInstructions_molecule        | 992     | +4/-4| 85.747  | 0.342    | 0.477       | 72.594             | 0.650     | 0.654      | 0.402 | 1.070 | 0.750|\n| Llama3.1-70B-Frags-Bio          | 991     | +4/-2| 85.379  | 0.343    | 0.477       | 72.171             | 0.649     | 0.653      | 0.401 | 1.075 | 0.751|\n| Llama3-8B-Generic               | 991     | +4/-5| 85.349  | 0.344    | 0.476       | 72.227             | 0.650     | 0.656      | 0.405 | 1.062 | 0.747|\n| Llama3-70B-Frags-Chem           | 990     | +2/-4| 85.295  | 0.345    | 0.478       | 72.278             | 0.651     | 0.656      | 0.403 | 1.070 | 0.750|\n| Llama3-70B-Frags-Quant          | 990     | +2/-3| 85.355  | 0.344    | 0.475       | 72.139             | 0.654     | 0.658      | 0.406 | 1.065 | 0.748|\n| Llama3-8B-Frags-Quant           | 990     | +2/-3| 85.375  | 0.344    | 0.476       | 72.171             | 0.655     | 0.660      | 0.407 | 1.064 | 0.748|\n| Llama3.1-8B-Quant               | 990     | +2/-5| 85.511  | 0.343    | 0.476       | 72.339             | 0.650     | 0.655      | 0.403 | 1.067 | 0.748|\n| Llama3-8B-Frags-Generic         | 989     | +2/-3| 85.271  | 0.347    | 0.477       | 72.035             | 0.652     | 0.658      | 0.404 | 1.069 | 0.749|\n| Llama3.1-70B-Frags-Drug         | 988     | +4/-1| 85.332  | 0.344    | 0.477       | 72.132             | 0.650     | 0.654      | 0.403 | 1.073 | 0.750|\n| Llama3-8B-Chem                  | 988     | +3/-3| 85.359  | 0.344    | 0.475       | 72.083             | 0.651     | 0.656      | 0.405 | 1.059 | 0.746|\n| Llama3-70B-Frags-Bio            | 986     | +3/-6| 85.389  | 0.343    | 0.477       | 72.168             | 0.650     | 0.653      | 0.402 | 1.074 | 0.751|\n| Llama3-8B-Bio                   | 986     | +3/-2| 85.456  | 0.343    | 0.474       | 72.265             | 0.650     | 0.656      | 0.405 | 1.064 | 0.746|\n| Gemma2-9B-Quant                 | 984     | +2/-2| 85.329  | 0.344    | 0.476       | 72.110             | 0.650     | 0.654      | 0.404 | 1.063 | 0.749|\n| Gemma2-9B-Drug                  | 984     | +3/-3| 85.266  | 0.344    | 0.477       | 72.262             | 0.649     | 0.654      | 0.403 | 1.068 | 0.750|\n| Gemma2-27B-Frags-Chem           | 983     | +1/-3| 85.319  | 0.345    | 0.478       | 72.136             | 0.647     | 0.651      | 0.400 | 1.078 | 0.753|\n| Llama3.1-70B-Frags-Quant        | 983     | +5/-2| 85.185  | 0.344    | 0.477       | 72.088             | 0.652     | 0.657      | 0.405 | 1.069 | 0.749|\n| Llama3.1-8B-Frags-Bio           | 982     | +2/-2| 85.322  | 0.343    | 0.479       | 72.117             | 0.647     | 0.651      | 0.399 | 1.079 | 0.753|\n| Llama3.1-8B-Bio                 | 982     | +1/-3| 85.431  | 0.344    | 0.476       | 72.155             | 0.649     | 0.654      | 0.403 | 1.064 | 0.748|\n| MistralNeMo-12B-Quant           | 981     | +3/-2| 85.404  | 0.343    | 0.478       | 72.092             | 0.649     | 0.654      | 0.403 | 1.068 | 0.751|\n| Llama3.1-8B-Frags-Quant         | 980     | +3/-3| 85.311  | 0.345    | 0.479       | 72.091             | 0.648     | 0.652      | 0.401 | 1.074 | 0.752|\n| MolT5_LPM24                     | 980     | +1/-4| 85.385  | 0.343    | 0.477       | 72.140             | 0.649     | 0.653      | 0.402 | 1.071 | 0.750|\n| Llama3.1-8B-Frags-Chem          | 978     | +1/-2| 85.340  | 0.344    | 0.478       | 72.132             | 0.647     | 0.652      | 0.400 | 1.078 | 0.752|\n| BlankCaption                    | 978     | +3/-2| 85.336  | 0.344    | 0.478       | 72.127             | 0.647     | 0.651      | 0.399 | 1.079 | 0.753|\n| Gemma2-27B-Frags-Quant          | 978     | +3/-1| 85.423  | 0.342    | 0.478       | 72.169             | 0.647     | 0.651      | 0.399 | 1.077 | 0.752|\n| Llama3.1-8B-Frags-Generic       | 977     | +5/-4| 85.326  | 0.346    | 0.478       | 72.101             | 0.648     | 0.653      | 0.401 | 1.074 | 0.751|\n| MistralNeMo-12B-Drug            | 975     | +5/-2| 85.323  | 0.345    | 0.477       | 72.097             | 0.650     | 0.654      | 0.402 | 1.070 | 0.750|\n| Gemma2-27B-Frags-Bio            | 973     | +2/-2| 85.431  | 0.343    | 0.478       | 72.274             | 0.647     | 0.651      | 0.399 | 1.079 | 0.753|\n| Gemma2-27B-Frags-Generic        | 973     | +1/-3| 85.362  | 0.342    | 0.478       | 72.032             | 0.647     | 0.652      | 0.399 | 1.078 | 0.753|\n| Llama3.1-8B-Frags-Drug          | 970     | +2/-1| 85.353  | 0.345    | 0.479       | 72.122             | 0.647     | 0.651      | 0.399 | 1.079 | 0.753|\n| GNN                             | NaN     | NaN    | 85.336  | 0.345    | 0.479       | 72.111             | 0.647     | 0.652      | 0.397 | 1.080 | 0.753   |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenentech%2Fmolcap-arena","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgenentech%2Fmolcap-arena","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenentech%2Fmolcap-arena/lists"}