{"id":16536340,"url":"https://github.com/timbmg/belief","last_synced_at":"2026-05-17T02:36:10.138Z","repository":{"id":37594898,"uuid":"147495010","full_name":"timbmg/belief","owner":"timbmg","description":"Implementation of Master Thesis on \"Belief State for Visually Grounded, Task-Oriented Neural Dialogue Model\"","archived":false,"fork":false,"pushed_at":"2024-07-25T10:48:10.000Z","size":25240,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T20:46:23.067Z","etag":null,"topics":["deep-learning","dialog","dialogue","dialogue-systems","machine-learning","neural-network","neural-networks","nlg","nlp","nlproc","pytorch","visual-grounding"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timbmg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-05T09:40:05.000Z","updated_at":"2022-06-22T11:25:28.000Z","dependencies_parsed_at":"2024-12-18T00:24:28.541Z","dependency_job_id":"36f91976-76d9-4567-b860-e8a4b8676827","html_url":"https://github.com/timbmg/belief","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/timbmg/belief","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2Fbelief","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2Fbelief/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2Fbelief/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2Fbelief/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timbmg","download_url":"https://codeload.github.com/timbmg/belief/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timbmg%2Fbelief/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259319536,"owners_count":22839790,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","dialog","dialogue","dialogue-systems","machine-learning","neural-network","neural-networks","nlg","nlp","nlproc","pytorch","visual-grounding"],"created_at":"2024-10-11T18:30:37.073Z","updated_at":"2025-10-23T17:02:32.868Z","avatar_url":"https://github.com/timbmg.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Belief State for Visually Grounded, Task-Oriented Neural Dialogue Model\nThis repository provides the code, [thesis](thesis.pdf) and [slides](slides.pdf) for my Master Thesis at University of Amsterdam on \"Belief State for Visually Grounded, Task-Oriented Neural Dialogue Model\".\n## Preparations\nThis code has been developed with python 3.6.5.\nRequirements can be found in the [requirements.txt](requirements.txt).\n\n### Downloads\n- Clone this repository.\n```bash\ngit clone git@github.com:timbmg/believe.git\n```\nNext, change into the guesswhat directory:\n```bash\ncd guesswhat\n```\n- Download the dataset, preprocessed VGG16 FC8 features and vocabulary files.\n```bash\nbash data/download.sh\n```\nAlternatively, you can create the VGG16 FC8 yourself with\n```bash\npython3 utils/cache_vgg_features.py\n```\nNote that you need the training and validation set of MS COCO 2014.  \nFor the Vocabulary and Category Vocabulary, if the files are not found in the specified folder for the training scripts, they will be created. Note that for the pretrained models, the provided Vocabulary and Category files need to be used.\n\n- Download the bin and log files of the pretrained models.\n```bash\nbash bin/download.sh\nbash analysis/download.sh\n```\n\n## Baseline Models\nFirst, the baseline models of the Oracle, Guesser and optionally Question Generator have to be trained. The default command line arguments are those of the original paper. The models can be trained with the following commands:\n- Oracle\n```bash\npython3 trainOracle.py\n```\n- Guesser\n```bash\npython3 trainGuesser.py\n```\n- Question Generator\n```bash\npython3 trainQGen.py\n```\n\n## Belief Models\n### Training\nThe different belief models can be trained with the following commands. Note that these are the settings that achieved the best results on the validation set. For all other hyperparameters, we refer to the help text in the training file.\n- Belief\n```bash\npython3 trainQGenBelief.py \\\n  --object-embedding-setting learn-emb-spatial \\\n  --visual-embedding-dim 0 \\\n  --category-embedding-dim 256\n```\n- Belief+FineTune\n```bash\npython3 trainQGenBelief.py \\\n  --train-guesser-setting \\\n  --object-embedding-setting learn-emb-spatial \\\n  --visual-embedding-dim 128 \\\n  --category-embedding-dim 512\n```\n- Visual Attention  \nFirst the ResNet features have to be cached. This can be done with the following script (warning: the file will be about 69GB large).\n```bash\npython3 cache_resnetblock3_features.py\n```\n```bash\npython3 trainQGenBelief.py \\\n  --train-guesser-setting \\\n  --visual-representation resnet-mlb \\\n  --object-embedding-setting learn-emb-spatial \\\n  --visual-embedding-dim 128 \\\n  --category-embedding-dim 512\n```\n\n### Evaluation\nAll Question Generator models can be evaluated with the inference script.\nIf the model is a belief model, the `-belief` option has to be passed additionally. In case the model shall be evaluated on the test set, please pass the `-split test` option. Further, in order to save the outputs (i.e. generated dialogues, belief and guesser probabilities), please pass `-save`. For all other options, we refer to the script.\n```bash\npython3 inference.py -bin/$qgenfile.pt\n```\n## Analysis\n### Stats and Graphs\nAll analysis can be reproduced from the [analysis jupyter notebook](guesswhat/analysis/analysis.ipynb). The log files are created by passing the `-save` option to the inference script or can be downloaded as mentioned in the Preparations section.\n### Web Tool\nIn order to run the web tool for comparing dialogues download the http-server from npm [here](https://www.npmjs.com/package/http-server).\nIf the log files are not in the analysis directory, move them there. Then change to the analysis directory and start the http-server.\n```bash\ncd analysis\nhttp-server\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimbmg%2Fbelief","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimbmg%2Fbelief","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimbmg%2Fbelief/lists"}