{"id":18439541,"url":"https://github.com/idiap/nvib_transformers","last_synced_at":"2025-04-15T03:43:01.472Z","repository":{"id":144962435,"uuid":"598198124","full_name":"idiap/nvib_transformers","owner":"idiap","description":null,"archived":false,"fork":false,"pushed_at":"2023-02-06T16:04:02.000Z","size":326,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-15T03:42:50.253Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idiap.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSES/GPL-3.0-only.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-06T15:56:52.000Z","updated_at":"2023-12-19T09:51:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"257b80e0-daaf-4a2f-90d8-477a0962c077","html_url":"https://github.com/idiap/nvib_transformers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fnvib_transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fnvib_transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fnvib_transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fnvib_transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idiap","download_url":"https://codeload.github.com/idiap/nvib_transformers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249003938,"owners_count":21196794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T06:25:20.316Z","updated_at":"2025-04-15T03:43:01.445Z","avatar_url":"https://github.com/idiap.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"..\n.. SPDX-FileCopyrightText: Copyright © 2023 Idiap Research Institute \u003ccontact@idiap.ch\u003e\n..\n.. SPDX-FileContributor: Fabio J Fehr \u003cfabio.fehr@idiap.ch\u003e\n..\n.. SPDX-License-Identifier: GPL-3.0-only\n..\n\n================================================================================================================\nA VAE for transformers using Nonparametric Information bottleneck\n================================================================================================================\n\n[Paper_]\n\n.. image:: figures/nvae.jpg\n\n\nThis repository contains the code developed for the paper: `A VAE for transformers using Nonparametric Information bottleneck`__ .\n\nPlease cite the original authors for their work in any publication(s) that uses this work:\n\n.. code:: bib\n\n    @inproceedings{henderson23_nvib,\n    author    = {James Henderson and Fabio Fehr},\n    title     = {{A VAE for Transformers with Nonparametric Variational Information Bottleneck}},\n    year      = {2023},\n    booktitle = {International Conference on Learning Representations},\n    url={https://openreview.net/forum?id=6QkjC_cs03X}\n    }\n\n\nInstallation\n------------\n\nTo create a new conda/mamba environment, install conda_, then mamba_, and simply follow the next steps:\n\n.. code:: bash\n\n    mamba env create -f environment.yml   # Create environment\n    mamba activate nvib                   # Activate environment\n\n*NB* This environment also installs the nvib package which is assumed to be in a directory one level above the current directory.\n\nCommand-line Usage\n-------------------\n\nData\n~~~~~~~~~~~~~~~~~~~~~~\nPrepare the data: Split into sentences, preprocessing, store the length distributions. Provide the local path for the\ndata, otherwise it will download directly from HuggingFace:\n\n\n.. code:: bash\n\n    python prepareDatasets.py --DATA wikitext2 --LOCAL_PATH None\n    python prepareDatasets.py --DATA wikitext103 --LOCAL_PATH None\n\nTraining\n~~~~~~~~~~~~~~~~~~~~~~\n\nFor the forward and reverse perplexity (F-PPL and R-PPL) we require an external langauge model we call\n\"vanillaTransformer\". Once this has been trained we train the model of choice. Logging is done with Weights and Biases (WandB).\nHowever, you may run it without logging on WandB.\n\n.. code:: bash\n\n    python train.py --EXPERIMENT_NAME vanillaTransformer --WANDB_ENTITY [WANDB_ENTITY]\n    python train.py --EXPERIMENT_NAME nvib --MODEL NVIB --WANDB_ENTITY [WANDB_ENTITY]\n\nReconstruction Evaluation\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nEvaluate the reconstruction of the model\n\n.. code:: bash\n\n    python reconstruction.py --EXPERIMENT_NAME nvib --WANDB_ENTITY [WANDB_ENTITY]\n\nDraw Samples from the prior\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nDraw samples from the model\n\n.. code:: bash\n\n    python sample.py --EXPERIMENT_NAME nvib --WANDB_ENTITY [WANDB_ENTITY]\n\n\nForward Perplexity Evaluation\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nEvaluate the forward perplexity of the model\n\n.. code:: bash\n\n    python forwardSampleEvaluation.py --EXPERIMENT_NAME nvib --LANGUAGE_MODEL vanillaTransformer --WANDB_ENTITY [WANDB_ENTITY]\n\n\nReverse Perplexity Evaluation\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nEvaluate the reverse perplexity of the model\n\n.. code:: bash\n\n    python reverseSampleEvaluation.py --EXPERIMENT_NAME nvib --LANGUAGE_MODEL vanillaTransformer --WANDB_ENTITY [WANDB_ENTITY]\n\n\nInterpolation Evaluation\n~~~~~~~~~~~~~~~~~~~~~~~~~~\nInterpolation between sentences evaluation\n\n.. code:: bash\n\n    python interpolation_evaluation.py --RUN_INTERPOLATIONS True --EXPERIMENT_NAME nvib --WANDB_ENTITY [WANDB_ENTITY]\n\n\nRepository Structure\n-----------------------------\n\n.. code:: bash\n\n    .\n    ├── classes\n    │   ├── Dataset.py\n    │   ├── MetricsAverage.py\n    │   ├── Transformer.py\n    │   ├── VariationalTransformerNVIB.py\n    │   ├── VariationalTransformerPooled.py\n    │   ├── VariationalTransformerStride.py\n    │   └── VariationalTransformerVariable.py\n    ├── data\n    │   └── \n    ├── environment.yml\n    ├── forwardSampleEvaluation.py\n    ├── interpolation_evaluation.py\n    ├── LICENSES\n    │   └── GPL-3.0-or-later.txt\n    ├── outputs\n    │   └── \n    ├── prepareDatasets.py\n    ├── README.rst\n    ├── reconstruction.py\n    ├── reverseSampleEvaluation.py\n    ├── sample.py\n    ├── train.py\n    └── utils.py\n\n\n\nContact\n---------\nFor questions or reporting issues to this software package, kindly contact the second author_.\n\n.. _author: fabio.fehr@idiap\n.. _Paper: https://openreview.net/forum?id=6QkjC_cs03X\n.. _conda: https://conda.io\n.. _mamba: https://mamba.readthedocs.io/en/latest/installation.html#existing-conda-install\n__ https://openreview.net/forum?id=6QkjC_cs03X\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Fnvib_transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidiap%2Fnvib_transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Fnvib_transformers/lists"}