{"id":16536304,"url":"https://github.com/timbmg/sentence-vae","last_synced_at":"2025-05-15T09:09:37.200Z","repository":{"id":43924621,"uuid":"123900776","full_name":"timbmg/Sentence-VAE","owner":"timbmg","description":"PyTorch Re-Implementation of \"Generating Sentences from a Continuous Space\" by Bowman et al 2015 https://arxiv.org/abs/1511.06349","archived":false,"fork":false,"pushed_at":"2025-01-14T10:04:01.000Z","size":252,"stargazers_count":591,"open_issues_count":12,"forks_count":154,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-14T15:56:54.133Z","etag":null,"topics":["deep-learning","generative-model","neural-network","nlp","ptb","pytorch","vae"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timbmg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"timbmg","ko_fi":"timbmg"}},"created_at":"2018-03-05T10:06:01.000Z","updated_at":"2025-03-18T10:48:27.000Z","dependencies_parsed_at":"2025-03-10T00:32:46.535Z","dependency_job_id":"a640f898-9580-4b5d-9646-baa6fb105ebe","html_url":"https://github.com/timbmg/Sentence-VAE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2FSentence-VAE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2FSentence-VAE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2FSentence-VAE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2FSentence-VAE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timbmg","download_url":"https://codeload.github.com/timbmg/Sentence-VAE/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254310520,"owners_count":22049470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","generative-model","neural-network","nlp","ptb","pytorch","vae"],"created_at":"2024-10-11T18:30:28.615Z","updated_at":"2025-05-15T09:09:32.191Z","avatar_url":"https://github.com/timbmg.png","language":"Python","funding_links":["https://github.com/sponsors/timbmg","https://ko-fi.com/timbmg"],"categories":[],"sub_categories":[],"readme":"# Sentence Variational Autoencoder\n\nPyTorch re-implementation of [_Generating Sentences from a Continuous Space_](https://arxiv.org/abs/1511.06349) by Bowman et al. 2015.\n![Model Architecture](https://github.com/timbmg/Sentence-VAE/blob/master/figs/model.png \"Model Architecture\")\n_Note: This implementation does not support LSTM's at the moment, but RNN's and GRU's._\n## Results\n### Training \n#### ELBO\n![ELBO](https://github.com/timbmg/Sentence-VAE/blob/master/figs/train_elbo.png \"ELBO\")\n#### Negative Log Likelihood\n![NLL](https://github.com/timbmg/Sentence-VAE/blob/master/figs/train_nll.png \"NLL\")\n### KL Divergence\n![KL](https://github.com/timbmg/Sentence-VAE/blob/master/figs/train_kl.png \"KL\")\n![KL Weight](https://github.com/timbmg/Sentence-VAE/blob/master/figs/kl_weight.png \"KL Weight\")\n\n\n### Performance\nTraining was stopped after 4 epochs. The true ELBO was optimized for approximately 1 epoch (as can bee see in the graph above). Results are averaged over entire split.\n\n| Split       | NLL   | KL    |\n|:------------|:------:|:-----:|\n| Train       | 99.821 | 7.944 |\n| Validation  | 103.220 | 7.346 |\n| Test        | 103.967 | 7.269 |\n### Samples\nSentenes have been obtained after sampling from z ~ N(0, I).  \n\n_mr . n who was n't n with his own staff and the n n n n n_  \n_in the n of the n of the u . s . companies are n't likely to be reached for comment_  \n_when they were n in the n and then they were n a n n_  \n_but the company said it will be n by the end of the n n and n n_  \n_but the company said that it will be n n of the u . s . economy_  \n\n### Interpolating Sentences\nSentenes have been obtained after sampling twice from z ~ N(0, I) and the interpolating the two samples.\n\n**the company said it will be n with the exception of the company**  \n_but the company said it will be n with the exception of the company ' s shares outstanding_  \n_but the company said that the company ' s n n and n n_  \n_but the company ' s n n in the past two years ago_  \n_but the company ' s n n in the past two years ago_  \n_but in the past few years ago that the company ' s n n_  \n_but in the past few years ago that they were n't disclosed_  \n_but in the past few years ago that they were n't disclosed_  \n_but in a statement that they were n't aware of the $ n million in the past few weeks_  \n**but in a statement that they were n't paid by the end of the past few weeks**  \n\n## Training\nTo run the training, please download the Penn Tree Bank data first (download from [Tomas Mikolov's webpage](http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz)). The code expects to find at least `ptb.train.txt` and `ptb.valid.txt` in the specified data directory. The data can also be donwloaded with the `dowloaddata.sh` script.\n\nThen training can be executed with the following command:\n```\npython3 train.py\n```\n\nThe following arguments are available:\n\n`--data_dir`  The path to the directory where PTB data is stored, and auxiliary data files will be stored.  \n`--create_data` If provided, new auxiliary data files will be created form the source data.  \n`--max_sequence_length` Specifies the cut off of long sentences.  \n`--min_occ` If a word occurs less than \"min_occ\" times in the corpus, it will be replaced by the \u003cunk\u003e token.  \n`--test` If provided, performance will also be measured on the test set.\n\n`-ep`, `--epochs`  \n`-bs`, `--batch_size`  \n`-lr`, `--learning_rate`\n\n`-eb`, `--embedding_size`  \n`-rnn`, `--rnn_type` Either 'rnn' or 'gru'.  \n`-hs`, `--hidden_size`  \n`-nl`, `--num_layers`  \n`-bi`, `--bidirectional`  \n`-ls`, `--latent_size`  \n`-wd`, `--word_dropout` Word dropout applied to the input of the Decoder, which means words will be replaced by `\u003cunk\u003e` with a probability of `word_dropout`.  \n`-ed`, `--embedding_dropout` Word embedding dropout applied to the input of the Decoder.\n\n`-af`, `--anneal_function` Either 'logistic' or 'linear'.  \n`-k`, `--k` Steepness of the logistic annealing function.  \n`-x0`, `--x0` For 'logistic', this is the mid-point (i.e. when the weight is 0.5); for 'linear' this is the denominator.\n\n`-v`, `--print_every`  \n`-tb`, `--tensorboard_logging` If provided, training progress is monitored with tensorboard.  \n`-log`, `--logdir` Directory of log files for tensorboard.  \n`-bin`,`--save_model_path` Directory where to store model checkpoints.\n\n## Inference\nFor obtaining samples and interpolating between senteces, inference.py can be used.\n```\npython3 inference.py -c $CHECKPOINT -n $NUM_SAMPLES\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimbmg%2Fsentence-vae","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimbmg%2Fsentence-vae","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimbmg%2Fsentence-vae/lists"}