{"id":21741213,"url":"https://github.com/imkett/adavae","last_synced_at":"2025-04-13T03:41:56.393Z","repository":{"id":63362859,"uuid":"441232675","full_name":"ImKeTT/AdaVAE","owner":"ImKeTT","description":"[Preprint] AdaVAE: Exploring Adaptive GPT-2s in VAEs for Language Modeling PyTorch Implementation","archived":false,"fork":false,"pushed_at":"2023-10-18T01:07:02.000Z","size":909,"stargazers_count":35,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-26T20:51:29.201Z","etag":null,"topics":["controllable-generation","gpt-2","nlp","parameter-efficient-tuning","representation-learning","text-classification","text-generation","vae","variational-autoencoder"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2205.05862","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ImKeTT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-23T16:11:22.000Z","updated_at":"2025-02-22T11:54:38.000Z","dependencies_parsed_at":"2022-11-17T16:31:09.379Z","dependency_job_id":null,"html_url":"https://github.com/ImKeTT/AdaVAE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImKeTT%2FAdaVAE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImKeTT%2FAdaVAE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImKeTT%2FAdaVAE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ImKeTT%2FAdaVAE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ImKeTT","download_url":"https://codeload.github.com/ImKeTT/AdaVAE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248660966,"owners_count":21141379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controllable-generation","gpt-2","nlp","parameter-efficient-tuning","representation-learning","text-classification","text-generation","vae","variational-autoencoder"],"created_at":"2024-11-26T06:17:23.808Z","updated_at":"2025-04-13T03:41:56.372Z","avatar_url":"https://github.com/ImKeTT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AdaVAE: Exploring Adaptive GPT-2s in VAEs for Language Modeling\n\n**[Repo In Progress]** Official implementation for **AdaVAE**, check the paper on arxiv [https://arxiv.org/pdf/2205.05862.pdf](https://arxiv.org/pdf/2205.05862.pdf).\n\n## Setup\n\nmake sure that you have installed:\n\n```\ntransformers==3.1.0\ntorch\ntensorboard\ntqdm\napex [from https://github.com/NVIDIA/apex]\nnltk\n```\n\n## Datasets\n\n- Language Modeling: yelp, yahoo, snli, ptb from [`download_datasets.md`](https://github.com/ChunyuanLI/Optimus/blob/master/data/download_datasets.md) in Optimus. Put them in the `./data/optimus_dataset` folder. \n- Low Resource Text Classification: yelp polarity dataset from [Shen et. al.](https://github.com/shentianxiao/language-style-transfer), and put it to `./data` folder directly. SST-2 and WNLI from GLUE, use [`download_glu_data.py`](https://github.com/ImKeTT/adavae/blob/main/low_nlu/download_glu_data.py) to download them, and put both datasets in the `./glue_data` folder.\n- Controllable Text Generation: yelp polarity dataset from [Shen et. al.](https://github.com/shentianxiao/language-style-transfer), and put it to `./data` folder directly.\n- Text Generation via Latent Manipulation: Any dataset mentioned above with pre-trained model weights in Language Modeling task.\n\nMake sure that all data folders contain `train.txt`, `test.txt`, `valid.txt` files.\n\n## Dependencies\n\n```\nadavae\n|____low_nlu\n| |____run_cls.sh\n| |____latent_classifier.py\n| |____utils_glue.py\n| |____...\n|____controlgen\n| |____oracle_cls.py\n| |____run.sh\n| |____run_vae_ctrl_gen.py\n| |____...\n|____README.md\n|____dialogue\n| |____run_spacefusion_gen.py\n| |____...\n|____data\n|____src\n| |____test.py\n| |____adapters\n| | |____vae.py\n| | |____...\n| |____adaVAE.py\n| |____run_manipulation.sh\n| |____run_lm.sh\n| |____test.py\n| |____...\n```\n\n## Tasks\n\n### Language Modeling\n\n\u003cimg src=\"model_LM.jpg\" alt=\"model_LM\" style=\"zoom:40%;\" /\u003e\n\nRun language modeling task by `bash src/run_lm.sh`, change arguments accordingly.\n\n### Low Resource Text Classification\n\n\u003cimg src=\"model_cls.jpg\" alt=\"model_cls\" style=\"zoom:30%;\" /\u003e\n\nRun classification task by `bash low_nlu/run_cls.sh`, change arguments accordingly.\n\n### Controllable Text Generation\n\nBefore conducting controllable text generation , you need to:\n\n1. Pre-train an oracle classifier for controllability evaluation by `python controlgen/oracle_cls.py`.\n2. Pre-train the AdaVAE model with Language Modeling task and load the weights.\n\nFinally run controllable text generation task by `bash controlgen/run.sh`, change arguments accordingly.\n\n### Text Generation via Latent Manipulation\n\nRun manipulation or analogy generation by `bash src/run_manipulation.sh`, the default dataset for this task if yelp polarity dataset. \n\n### Visualization\n\nTBD.\n\n### Dialog Generation\n\nTBD.\n\n### Model Testing\n\nTBD. [Check `src/test.py` for more information]\n\n\n\n## Others\n\nPlease email me or open an issue if you have any question.\n\nif you find our work useful, please cite the paper :\u003e\n\n```\n@article{tu2022adavae,\n  title={AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling},\n  author={Tu, Haoqin and Yang, Zhongliang and Yang, Jinshuai and Zhang, Siyu and Huang, Yongfeng},\n  journal={arXiv preprint arXiv:2205.05862},\n  year={2022}\n}\n```\n\nWe thank open sourced codes related to VAEs and parameter-efficient methods, which inspired our work !!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimkett%2Fadavae","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimkett%2Fadavae","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimkett%2Fadavae/lists"}