{"id":27944874,"url":"https://github.com/devslem/mol-air","last_synced_at":"2025-05-07T12:54:58.356Z","repository":{"id":225758033,"uuid":"757789983","full_name":"DevSlem/Mol-AIR","owner":"DevSlem","description":"Molecular Reinforcement Learning with Adaptive Intrinsic Reward for Goal-directed Molecular Generation.","archived":false,"fork":false,"pushed_at":"2025-02-27T01:44:09.000Z","size":12318,"stargazers_count":15,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-07T12:54:50.158Z","etag":null,"topics":["chemistry","deep-reinforcement-learning","drug-discovery","molecular-modeling","molecular-structure","molecules","reinforcement-learning"],"latest_commit_sha":null,"homepage":"https://pubs.acs.org/doi/10.1021/acs.jcim.4c01669","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DevSlem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-15T01:28:25.000Z","updated_at":"2025-05-01T09:43:07.000Z","dependencies_parsed_at":"2024-03-19T11:49:59.680Z","dependency_job_id":"23e4830c-a637-4b9c-8acc-a131e560d7b7","html_url":"https://github.com/DevSlem/Mol-AIR","commit_stats":null,"previous_names":["devslem/mol-air"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevSlem%2FMol-AIR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevSlem%2FMol-AIR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevSlem%2FMol-AIR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevSlem%2FMol-AIR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DevSlem","download_url":"https://codeload.github.com/DevSlem/Mol-AIR/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252882791,"owners_count":21819154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chemistry","deep-reinforcement-learning","drug-discovery","molecular-modeling","molecular-structure","molecules","reinforcement-learning"],"created_at":"2025-05-07T12:54:54.334Z","updated_at":"2025-05-07T12:54:58.354Z","avatar_url":"https://github.com/DevSlem.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mol-AIR\n\n**Mol-AIR** is a reinforcement learning-based framework using adaptive intrinsic rewards for effective goal-directed molecular generation. Mol-AIR leverages the strengths of **both history-based and learning-based intrinsic rewards** by exploiting random distillation network and counting-based strategies. You can find the details of Mol-AIR in the paper: [\"Mol-AIR: Molecular Reinforcement Learning with Adaptive Intrinsic Rewards for Goal-Directed Molecular Generation\"](https://pubs.acs.org/doi/10.1021/acs.jcim.4c01669).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"img/toc.png\" width=\"70%\"\u003e\n\u003c/p\u003e\n\nThis repository provides the implementation of Mol-AIR and the experiments on the molecular generation tasks.\n\n| [Overview](#overview) | [Setup \u0026 Run Experiments](#setup--run-experiments) | [Experiment Results](#experiment-results) | [Configuration](#configuration) | [Citation](#citation) |\n\n## Overview\n\nWe have two types of experiments: **pure RL** and **pre-training + RL**. \n\n### Pure RL Experiments\n\nThe pure RL experiments show that our proposed intrinsic reward method, **Mol-AIR**, can help the agent to explore the chemical space effectively so that the agent can find better molecules. The experiments were conducted on the 6 target properties: **pLogP, QED, Similarity, GSK3B, JNK3, and GSK3B+JNK3 (two objectives)**. The below figure shows Mol-AIR adaptively balances between exploration and exploitation during training:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"img/fig4.png\" width=\"70%\"\u003e\n\u003c/p\u003e\n\n### Pre-training + RL Experiments\n\nThe pre-training + RL experiments show that Mol-AIR works well in optimizing the pre-trained model to generate the various molecules with the desired properties. The experiments were conducted on the **DRD2+QED+SA** property (three objectives). The primary objective of the experiments is to discover dopamine receptor D2 (DRD2) inhibitors with high quantitative estimate of drug-likeness (QED) and low synthetic accessibility (SA) score. The below figure shows that our method successfully\ngenerated compounds structurally similar to risperidone, a well-known DRD2 inhibitor:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"img/fig7.png\" width=\"70%\"\u003e\n\u003c/p\u003e\n\n## Setup \u0026 Run Experiments\n\nThe pure RL experiments were conducted with 'SELFIES 0.2.4' for comparison with the previous methods. While the pre-training + RL experiments were conducted with 'SELFIES 2.1.2' since 'SELFIES 0.2.4' is too old to construct the vocabulary for training. Therefore, we highly recommend to install the desired version of SELFIES to reproduce our experiments, even if it is compatible with any version and customizable. Also, we recommend to use conda to create a new environment for each experiment type.\n\n### Pure RL Experiments\n\nIf you use conda, create a new Python 3.7.16 environment:\n\n```bash\nconda create -n mol-air python=3.7.16 -y\nconda activate mol-air\n```\n\n\u003e Note: If you don't use Python 3.7, you may have dependency issues.\n\nInstall the required packages using pip (SELFIES 0.2.4 is included):\n\n```bash\npip install -r requirements.txt\n```\n\n\u003e Note: The [requirements.txt](requirements.txt)  was tested only on Ubuntu (Linux) OS.\n\nAll experiment configurations are in the [config/](config/) directory. `ppo.yaml` is the configuration for vanilla PPO, `hir.yaml` is for HIR (history-based intrinsic reward), and `lir.yaml` is for LIR (learning-based intrinsic reward) and `molair.yaml` is for Mol-AIR (both history-based and learning-based intrinsic rewards). You can run the experiments by entering the below command:\n\n```bash\npython run.py [CONFIG_PATH]\n```\n\nFor example, pLogP with PPO is `$ python run.py config/plogp/ppo.yaml` and pLogP with Mol-AIR is `$ python run.py config/plogp/molair.yaml`:\n\n```bash\npython run.py config/plogp/molair.yaml # pLogP\npython run.py config/qed/molair.yaml # QED\npython run.py config/similarity/molair.yaml # Similarity\npython run.py config/gsk3b/molair.yaml # GSK3B\npython run.py config/jnk3/molair.yaml # JNK3\npython run.py config/gsk3b+jnk3/molair.yaml # GSK3B+JNK3\n```\n\n### Pre-training + RL Experiments (Generative Model)\n\nWe recommend to create a different environment from the pure RL experiments for the pre-training + RL experiments:\n\n```bash\nconda create -n mol-air-gen python=3.7.16 -y\nconda activate mol-air-gen\n```\n\n\u003e Note: If you don't use Python 3.7, you may have dependency issues.\n\nInstall the required packages using pip (SELFIES 2.1.2 is included):\n\n```bash\npip install -r requirements_gen.txt\n```\n\n\u003e Note: The [requirements_gen.txt](requirements_gen.txt) was tested only on Ubuntu (Linux) OS.\n\nRun the end-to-end experiment of the DRD2+QED+SA property from pre-training to Mol-AIR by entering the below commands ([molair_end2end.yaml](config/drd2+qed+sa/molair_end2end.yaml)):\n\n```bash\npython run.py config/drd2+qed+sa/molair_end2end.yaml -p # Pre-training\npython run.py config/drd2+qed+sa/molair_end2end.yaml # RL training\npython run.py config/drd2+qed+sa/molair_end2end.yaml -i # Inference\n```\n\nwhere \"Inference\" is to generate the molecules with the trained agent. Alternatively, you can skip the pre-training stage by using the provided [pre-trained model](data/drd2+qed+sa/pretrained.pt) and [vocabulary](data/drd2+qed+sa/vocab.json). In this case, just run the Mol-AIR experiment by entering the below commands ([molair.yaml](config/drd2+qed+sa/molair.yaml)):\n\n```bash\npython run.py config/drd2+qed+sa/molair.yaml # RL training\npython run.py config/drd2+qed+sa/molair.yaml -i # Inference\n```\n\nAlso, you can run the PPO experiment by ([ppo.yaml](config/drd2+qed+sa/ppo.yaml)):\n\n```bash\npython run.py config/drd2+qed+sa/ppo.yaml # RL training\npython run.py config/drd2+qed+sa/ppo.yaml -i # Inference\n```\n\nand the experiment of the pre-trained agent by ([pretrained.yaml](config/drd2+qed+sa/pretrained.yaml)):\n\n```bash\npython run.py config/drd2+qed+sa/pretrained.yaml # Just wrapping\npython run.py config/drd2+qed+sa/pretrained.yaml -i # Inference\n```\n\n\u003e Note: RL training with the pre-trained agent configuration ([pretrained.yaml](config/drd2+qed+sa/pretrained.yaml)) is just wrapping the pre-trained model with the `PretrainedRecurrentAgent` class for the compatibility with the RL agent. The pre-trained model is never updated by the command.\n\n## Experiment Results\n\nAll experiment results are saved in the `results/[EXPERIMENT_ID]/` directory. For example, if you run the pLogP with Mol-AIR, the result is saved in `results/PLogP-MolAIR`. You can see the training plots of each experiment in the `results/[EXPERIMENT_ID]/plots/` directory. Also, you can see the training plots at once by TensorBoard: `$ tensorboard --logdir=results`. This makes you easily compare the results of different experiments.\n\n### RL Training\n\nThe training plots are saved in the directories:\n\n* `plots/Environment/`: metrics of the training environments.\n* `plots/Inference/`: metrics of the inference (validation) environments during training.\n* `plots/Training/`: losses of the RL agent.\n\nThe RL agents are saved as:\n\n* `agent.pt`: final agent.\n* `best_agent.pt`: best agent which has the highest score from the inference environments during training.\n* `agent_ckpt/`: agent checkpoints directory at the every `agent_save_freq` frequency.\n\n\u003e Note: The pre-trained agent has only `agent.pt`.\n\nThe most important file is `episode_metric.csv` which contains the metrics of the generated molecules at each episode during training.\n\n### Inference\n\nThe inference results are saved in the directory: `inference/`. You can see the three or four files in this directory:\n\n* `metrics.csv`: summary of the metrics of the generated molecules.\n* `molecules.csv`: generated molecules with the metrics.\n* `best_molecule.csv`: the best molecule with the highest score.\n* `top_50_unique_molecules.png` (optional): images of the top 50 unique molecules with the highest scores.\n\n\u003e Note: `top_50_unique_molecules.png` requires the `libXrender.so.1`. If you want to draw molecules, you should install it by `$ sudo apt-get install libxrender1` or `$ conda install -c conda-forge libxrender`.\n\n### Pre-training\n\nThe pre-training plots are saved in the `plots/Pretrain/` directory. The pre-trained models are saved in the `pretrained_models/` directory:\n\n* `best.pt`: best pre-trained model which has the lowest validation loss.\n* `final.pt`: final pre-trained model at the end of the pre-training.\n\nAlso, the vocabulary used in the pre-training is saved as the `vocab.json` file.\n\n## Configuration\n\nRefer to the [Mol-AIR Experiment Configuration](config/README.md) document for the detailed configuration of the experiments.\n\n## Citation\n\nPlease cite our work if you find it useful:\n\n```\n@article{park2025mol,\n  title={Mol-AIR: Molecular Reinforcement Learning with Adaptive Intrinsic Rewards for Goal-directed Molecular Generation},\n  author={Park, Jinyeong and Ahn, Jaegyoon and Choi, Jonghwan and Kim, Jibum},\n  journal={Journal of Chemical Information and Modeling},\n  year={2025},\n  publisher={ACS Publications}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevslem%2Fmol-air","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevslem%2Fmol-air","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevslem%2Fmol-air/lists"}