{"id":19085045,"url":"https://github.com/jaywalnut310/vector-quantized-autoencoders","last_synced_at":"2025-04-30T09:26:10.041Z","repository":{"id":49755173,"uuid":"171448690","full_name":"jaywalnut310/Vector-Quantized-Autoencoders","owner":"jaywalnut310","description":"Tensorflow Implementation of \"Theory and Experiments on Vector Quantized Autoencoders\"","archived":false,"fork":false,"pushed_at":"2019-02-27T06:00:46.000Z","size":509,"stargazers_count":14,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-30T14:51:12.057Z","etag":null,"topics":["autoencoder","deep-learning","tensorflow","transformer","vae","vqvae","wmt"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaywalnut310.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-02-19T09:53:49.000Z","updated_at":"2024-05-07T22:50:49.000Z","dependencies_parsed_at":"2022-09-24T05:41:43.065Z","dependency_job_id":null,"html_url":"https://github.com/jaywalnut310/Vector-Quantized-Autoencoders","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaywalnut310%2FVector-Quantized-Autoencoders","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaywalnut310%2FVector-Quantized-Autoencoders/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaywalnut310%2FVector-Quantized-Autoencoders/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaywalnut310%2FVector-Quantized-Autoencoders/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaywalnut310","download_url":"https://codeload.github.com/jaywalnut310/Vector-Quantized-Autoencoders/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251675771,"owners_count":21625886,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoencoder","deep-learning","tensorflow","transformer","vae","vqvae","wmt"],"created_at":"2024-11-09T02:53:37.601Z","updated_at":"2025-04-30T09:26:10.016Z","avatar_url":"https://github.com/jaywalnut310.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Vector Quantized Autoencoders\n\n## WARNING: README is not written well currently. I'll clean it in a few weeks!\n\nTensorflow implementation of [Theory and Experiments on Vector Quantized Autoencoders](https://arxiv.org/abs/1805.11063).\n\nBy modifying configurations, you can use VQVAE instead of soft EM version VQA (modify bottleneck_kind to vq in [config.yml](config.yml))\n\nFor more details, please refer to the paper or its precedent paper ([Neural Discrete Representation Learning](https://arxiv.org/abs/1711.00937)).\n\n## Notes\n* Many codes of this repository are drawn from [tensor2tensor](https://github.com/tensorflow/tensor2tensor) library.\n* As tensor2tensor is too big to understand at a glance, I draw some of their codes as concise as possible.\n* All works are done in TensorFlow 1.12.\n\n\n## WMT14-ENDE (Distilled Data)\nI got 24.9 BLEU score. It's quite not that bad, but still worse than the paper's result.\n\nI trained with configurations as below:\n* 4 V100 GPUs\n* batch_size: 8192\n* Knowledge Distillation from [Transformer of OpenNMT-tf](https://github.com/OpenNMT/OpenNMT-tf/tree/master/scripts/wmt)\n* averages weights of recent checkpoints using [avg_checkpoints.py](avg_checkpoints.py)\n\nIt took about 13 days to run 1M train steps.\n\nIn my experience, I figured more batch_size and number of gpus help to improve performance significantly.\n\nThus, there is a possiblility to get further improvements with 8 or more gpus training.\n\nIf you have enough gpus, please let me know the result.\n\nAdditionally, if there is any error, mis-implementation or mis-configuration, please let me know :).\n\n\n### Data Preparation\nTK\n```sh\n# You can change filepaths by modifying 'config.yml'\n# or you can change filepaths using --c config as below.\n# python generate_data.py --c \\\n#   source_vocab_file=/path/to/your-data-dir/vocab_src \\\n#   target_vocab_file=/path/to/your-data-dir/vocab_tgt \\\n#   source_train_file=/path/to/your-data-dir/train_src \\\n#   target_train_file=/path/to/your-data-dir/train_tgt \\\n#   source_eval_file=/path/to/your-data-dir/valid_src \\\n#   target_eval_file=/path/to/your-data-dir/valid_tgt \\\n#   record_train_file=/path/to/your-data-dir/train.tfrecords \\\n#   record_eval_file=/path/to/your-data-dir/valid.tfrecords\npython generate_data.py\n```\n\n### Training\n```sh\n# 1. Create log directory\nmkdir /path/to/your-log-dir\n\n# 2. (Optional) Copy configs\ncp ./config.yml /path/to/your-log-dir\n\n# 3. Run training\npython train.py -m /path/to/your-log-dir\n```\n\nIf you want to change hparams, then you can do it by choosing one of two options.\n* modify [config.yml](config.yml)\n* add arguments as below:\n  ```sh\n  python train.py -m /path/to/your-log-dir --c hidden_size=512 num_heads=8\n  ```\n\n### Test\nTK\n```sh\n# (Optional) Averaging checkpoints is mostly helpful to improve performance\npython avg_checkpoints.py --prefix=/path/to/your-log-dir --num_last_checkpoints=20\n\n# checkpoint config is optional\npython decode.py \\\n  --model_dir /path/to/your-log-dir \\\n  --predict_file /path/to/wmt14_ende_distill/test.en \\\n  --out_file out.txt\n  --checkpoint /tmp/averaged.ckpt-0\n\nspm_decode \\\n  --model=/path/to/OpenNMT-tf/scripts/wmt/wmtende.model \\\n  --input_format=piece \u003c out.txt \u003e out.detok.txt\n\nsh /path/to/OpenNMT-tf/scripts/wmt/get_ende_bleu.sh out.detok.txt\n```\n\nCurrent result:\n```\nBLEU = 24.89, 57.6/31.2/19.1/12.2 (BP=0.978, ratio=0.978, hyp_len=63093, ref_len=64496)\n```\n\n#### Prediction samples\n| Source        | Prediction    | Ground Truth  |\n| ------------- | ------------- | ------------- |\n| Gutach: Increased safety for pedestrians | Guts: Mehr Sicherheit für Fußgänger | Gutach: Noch mehr Sicherheit für Fußgänger |\n| They are not even 100 metres apart: On Tuesday, the new B 33 pedestrian lights in Dorfparkplatz in Gutach became operational - within view of the existing Town Hall traffic lights. | Sie sind nicht nicht einmal 100 Meter voneinander entfernt: Am Dienstag wurden die neuen Fußgängerzonen B 33 am Dorfparkplatz in Gutach im Hinblick auf die bestehende Ampel des Rathauses in Betrieb genommen. | Sie stehen keine 100 Meter voneinander entfernt: Am Dienstag ist in Gutach die neue B 33-Fußgängerampel am Dorfparkplatz in Betrieb genommen worden - in Sichtweite der älteren Rathausampel. |\n| Two sets of lights so close to one another: intentional or just a silly error? | Zwei Lichtsätze so nah aneinander: absichtlich oder nur ein dummer Fehler? | Zwei Anlagen so nah beieinander: Absicht oder Schildbürgerstreich? |\n\nThe above samples are just the first 3 sentences in the test set.\nMore samples can be found in [source.txt](resources/source.txt), [prediction.txt](resources/prediction.txt) and [ground_truth.txt](resources/ground_truth.txt).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaywalnut310%2Fvector-quantized-autoencoders","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaywalnut310%2Fvector-quantized-autoencoders","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaywalnut310%2Fvector-quantized-autoencoders/lists"}