{"id":20028143,"url":"https://github.com/sominw/vqamd_floyd","last_synced_at":"2025-05-05T03:32:03.525Z","repository":{"id":37070072,"uuid":"141307780","full_name":"sominw/vqamd_floyd","owner":"sominw","description":"Visual Question Answering through modal dialogue + API ","archived":false,"fork":false,"pushed_at":"2022-12-08T00:45:33.000Z","size":20911,"stargazers_count":15,"open_issues_count":16,"forks_count":9,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-08T16:04:51.831Z","etag":null,"topics":["deep-learning","deep-neural-networks","floydhub","machine-learning","visual-question-answering"],"latest_commit_sha":null,"homepage":"https://blog.floydhub.com/asking-questions-to-images-with-deep-learning/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sominw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-17T15:19:03.000Z","updated_at":"2022-08-19T16:18:44.000Z","dependencies_parsed_at":"2023-01-25T01:45:16.006Z","dependency_job_id":null,"html_url":"https://github.com/sominw/vqamd_floyd","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sominw%2Fvqamd_floyd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sominw%2Fvqamd_floyd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sominw%2Fvqamd_floyd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sominw%2Fvqamd_floyd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sominw","download_url":"https://codeload.github.com/sominw/vqamd_floyd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252434934,"owners_count":21747321,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deep-neural-networks","floydhub","machine-learning","visual-question-answering"],"created_at":"2024-11-13T09:13:35.064Z","updated_at":"2025-05-05T03:31:58.835Z","avatar_url":"https://github.com/sominw.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"  # Visual Question Answering through Modal Dialogue\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/sominwadhwa/sominwadhwa.github.io/blob/master/assets/vqa/1.jpeg?raw=true\"/\u003e\n\u003c/p\u003e\n\nWe’re already seeing incredible applications of object detection in our daily lives. One such interesting application is Visual Question Answering. It is a new and upcoming problem in Computer Vision where the data consists of open-ended questions about images. In order to answer these questions, an effective system would need to have an understanding of  “[vision, language and common-sense](https://dac.cs.vt.edu/research-project/visual-question-answering-vqa/).”\n\nBefore proceeding further, I would highly encourge you to quickly read the full VQA Post here. \n\n## Try it now on FloydHub\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://floydhub.com/run\"\u003e\n    \u003cimg src=\"https://static.floydhub.com/button/button.svg\" alt=\"Run\"\u003e\n\u003c/a\u003e\n\u003c/p\u003e\n\nClick this button to open a Workspace on FloydHub that will train this model.\n\nDo remember to execute **`run_me_first_floyd.sh` inside a terminal everytime** you restart your workspace to install relevant dependencies. \n\u003c!---\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/sominwadhwa/sominwadhwa.github.io/blob/master/assets/vqa/9.png?raw=true\"/\u003e\n\u003c/p\u003e\n--\u003e\n---\n\nThis post will first dig into the basic theory behind the Visual Question Answering task. Then, we’ll discuss and build two approaches to VQA: the “bag-of-words” and the “recurrent” model. Finally, we’ll provide a tutorial workflow for training your own models and setting up a REST API on FloydHub to start detecting objects in your own images.\nThe project code is in Python (Keras + TensorFlow). You can view my experiments directly on [FloydHub](https://www.floydhub.com/sominw/projects/vqa_floyd), as well as the code (along with the weight files and data) on [Github](https://github.com/sominwadhwa/vqamd_floyd).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/sominwadhwa/sominwadhwa.github.io/blob/master/assets/vqa/8.gif?raw=true\"/\u003e\n\u003c/p\u003e\n\nSince I've already preprocessed the data \u0026 stored everything in a FloydHub [dataset](https://www.floydhub.com/sominw/datasets/vqa_data), here's what we're going to do - \n\n- Checkout the preprocessed data from the VQA Dataset.\n- Build \u0026 train **two** `VQA` models using Keras \u0026 Tensorflow.\n- Assess the models on the `VQA` validation sets.\n- Run the model to generate some really cool predictions. \n\n## Serving Models on FloydHub\n\nI've created a separate [repository here](https://github.com/sominwadhwa/vqamd_api) to serve models since it **avoids the overhead of pushing the entire code/data in the training repo to Floyd over \u0026 over again**. \n\n---\n\n## For Offline Execution\n\nThe following are a couple of instructions that must be gone through in order to execute different (or all) sections of this project. You will need a **NVIDIA GPU** to train these models.\n\n1. Clone the project, replacing ``VQAMD`` with the name of the directory you are creating:\n\n        $ git clone https://github.com/sominwadhwa/vqa_floyd.git VQAMD\n        $ cd VQAMD\n\n2. Make sure you have ``python 3.5.x`` running on your local system. If you do, skip this step. In case you don't, head\nhead [here](https://www.python.org/downloads/).\n\n3. ``virtualenv`` is a tool used for creating isolated 'virtual' python environments. It is advisable to create one here as well (to avoid installing the pre-requisites into the system-root). Do the following within the project directory:\n\n        $ [sudo] pip install virtualenv\n        $ virtualenv --system-site-packages VQAMD\n        $ source VQAMD/bin/activate\n\nTo deactivate later, once you're done with the project, just type ``deactivate``.\n\n4. Install the pre-requisites from ``requirements.txt`` \u0026 run ``tests/init.py`` to check if all the required packages were correctly installed:\n\n        $ pip install -r requirements.txt\n        $ bash run_me_first_on_floyd.sh\n\n## Contributing to VQA\n\nI welcome contributions to this little project. If you have any new ideas or approaches that you'd like to incorporate here, feel free to open up an issue.\n\nPlease refer to each project's style guidelines and guidelines for submitting patches and additions. In general, we follow the \"fork-and-pull\" Git workflow.\n\n 1. **Fork** the repo VQAMD on GitHub\n 2. **Clone** the project to your own machine\n 3. **Commit** changes to your own branch\n 4. **Push** your work back up to your fork\n 5. Submit a **Pull request** so that we can review your changes\n\nNOTE: Be sure to merge the latest from \"upstream\" before making a pull request!\n\n### Issues\n\nFeel free to submit issues and enhancement requests.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsominw%2Fvqamd_floyd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsominw%2Fvqamd_floyd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsominw%2Fvqamd_floyd/lists"}