{"id":17545585,"url":"https://github.com/haoliuhl/language-quantized-autoencoders","last_synced_at":"2025-03-16T13:31:10.808Z","repository":{"id":65898608,"uuid":"596869278","full_name":"haoliuhl/language-quantized-autoencoders","owner":"haoliuhl","description":"Language Quantized AutoEncoders","archived":false,"fork":false,"pushed_at":"2023-02-07T19:15:29.000Z","size":38,"stargazers_count":101,"open_issues_count":2,"forks_count":5,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-16T02:39:42.371Z","etag":null,"topics":["autoencoders","bert","large-language-models","multimodal","roberta","vq","vqvae"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/haoliuhl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-02-03T05:11:48.000Z","updated_at":"2025-03-09T03:21:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"faadeac2-8cb3-4df9-ac26-32852c9fec84","html_url":"https://github.com/haoliuhl/language-quantized-autoencoders","commit_stats":null,"previous_names":["lhao499/language-quantized-autoencoders","forhaoliu/language-quantized-autoencoders","haoliuhl/language-quantized-autoencoders","lhao499/lqae"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haoliuhl%2Flanguage-quantized-autoencoders","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haoliuhl%2Flanguage-quantized-autoencoders/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haoliuhl%2Flanguage-quantized-autoencoders/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haoliuhl%2Flanguage-quantized-autoencoders/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/haoliuhl","download_url":"https://codeload.github.com/haoliuhl/language-quantized-autoencoders/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243875106,"owners_count":20361947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoencoders","bert","large-language-models","multimodal","roberta","vq","vqvae"],"created_at":"2024-10-21T01:01:39.975Z","updated_at":"2025-03-16T13:31:10.556Z","avatar_url":"https://github.com/haoliuhl.png","language":"Python","funding_links":[],"categories":["NLP"],"sub_categories":[],"readme":"# Language Quantized AutoEncoders\n\nThis is a Jax implementation of our work [Language Quantized AutoEncoders](https://arxiv.org/abs/2302.00902).\n\nIt contains training and evalutation code.\n\nThis implementation has been tested on multi-GPU and Google Cloud TPU and supports both multi-host training with TPU Pods and multi-GPU training.\n\n## Usage\nExperiments can be launched via the following commands.\n\nAn example script of launching a LQAE training job is:\n```\nexport SCRIPT_DIR=\"$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" \u0026\u003e /dev/null \u0026\u0026 pwd )\"\nexport PROJECT_DIR=\"$( cd -- \"$( dirname -- \"$SCRIPT_DIR\" )\" \u0026\u003e /dev/null \u0026\u0026 pwd )\"\ncd $PROJECT_DIR\nexport PYTHONPATH=\"$PYTHONPATH:$PROJECT_DIR\"\n\necho $PYTHONPATH\nexport WANDB_API_KEY=''\n\nexport experiment_name='lqae'\nexport project_id='lqae'\nexport wu='5'\nexport ep='100'\nexport model='lqae'\nexport experiment_note=\"\"\nexport experiment_id=\"lqae-base\"\n\npython3 -m lqae.main.lqae_main \\\n    --model_type=\"$model\" \\\n    --lqae.bert_min_ratio=0.5 \\\n    --lqae.bert_max_ratio=0.5 \\\n    --lqae.quantizer_loss_commitment=0.005 \\\n    --lqae.quantizer_loss_entropy=0.0 \\\n    --lqae.quantizer_loss_perplexity=0.0 \\\n    --lqae.l2_normalize=True \\\n    --lqae.top_k_value=1 \\\n    --lqae.top_k_avg=False \\\n    --lqae.top_k_rnd=False \\\n    --lqae.vit_encoder_decoder=True \\\n    --lqae.vit_model_type='base' \\\n    --lqae.patch_size=16 \\\n    --lqae.use_bert_codebook=True \\\n    --lqae.bert_mask_loss_weight=0.0001 \\\n    --lqae.bert_channel_image_loss_weight=1.0 \\\n    --lqae.nochannel_image_loss_weight=0.0 \\\n    --lqae.quantizer_latent_dim=0 \\\n    --lqae.strawman_codebook=False \\\n    --lqae.use_bert_ste=False \\\n    --seed=42 \\\n    --epochs=\"$ep\" \\\n    --lr_warmup_epochs=\"$wu\" \\\n    --batch_size=512 \\\n    --dataloader_n_workers=16 \\\n    --log_freq=500 \\\n    --plot_freq=2000 \\\n    --save_model_freq=10000 \\\n    --lr_peak_value=1.5e-4 \\\n    --weight_decay=0.0005 \\\n    --load_checkpoint='' \\\n    --dataset='imagenet' \\\n    --imagenet_data.path=\"YOUR IMAGENET FILE in HDF5\" \\\n    --imagenet_data.random_start=True \\\n    --log_all_worker=False \\\n    --logging.online=True \\\n    --logging.project_id=\"$project_id\" \\\n    --logging.experiment_id=\"$experiment_id\" \\\n    --logging.experiment_note=\"$experiment_note\" \\\n    --logging.output_dir=\"$HOME/experiment_output/$project_id\"\n\n```\n\nExample of running LLM based evaluation using LQAE pretrained model is at this [colab](https://colab.research.google.com/drive/1_nzC8W6yO9fYP8GLfUmY11hoVQUW9e6Q?usp=sharing).\n\nTo run experiments more conveniently on TPUs, you may want to use the script in jobs folder to manage TPUs jobs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaoliuhl%2Flanguage-quantized-autoencoders","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhaoliuhl%2Flanguage-quantized-autoencoders","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaoliuhl%2Flanguage-quantized-autoencoders/lists"}