{"id":32929217,"url":"https://github.com/msmrexe/neurosymbolic-vqa-program-generator","last_synced_at":"2026-05-07T09:31:18.004Z","repository":{"id":322245433,"uuid":"1088717234","full_name":"msmrexe/neurosymbolic-vqa-program-generator","owner":"msmrexe","description":"A comprehensive implementation of a Neurosymbolic framework for Visual Question Answering (VQA) on the CLEVR dataset. This project translates natural language questions into symbolic programs using three different learning strategies: Supervised (LSTM \u0026 Transformer), Reinforcement Learning (REINFORCE), and In-Context Learning (LLM).","archived":false,"fork":false,"pushed_at":"2025-11-03T12:49:54.000Z","size":74,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-03T13:07:01.445Z","etag":null,"topics":["clevr","course-project","in-context-learning","large-language-models","lstm","neurosymbolic","neurosymbolic-ai","policy-gradient","program-generator","pytorch","reinforce","reinforcement-learning","seq2seq","supervised-learning","system-2","transformer","university-project","visual-question-answering","visual-reasoning","vqa"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/msmrexe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-03T11:08:26.000Z","updated_at":"2025-11-03T12:55:36.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/msmrexe/neurosymbolic-vqa-program-generator","commit_stats":null,"previous_names":["msmrexe/neurosymbolic-vqa-program-generator"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/msmrexe/neurosymbolic-vqa-program-generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msmrexe%2Fneurosymbolic-vqa-program-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msmrexe%2Fneurosymbolic-vqa-program-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msmrexe%2Fneurosymbolic-vqa-program-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msmrexe%2Fneurosymbolic-vqa-program-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/msmrexe","download_url":"https://codeload.github.com/msmrexe/neurosymbolic-vqa-program-generator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/msmrexe%2Fneurosymbolic-vqa-program-generator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32731196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T02:14:30.463Z","status":"ssl_error","status_checked_at":"2026-05-07T02:14:29.405Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clevr","course-project","in-context-learning","large-language-models","lstm","neurosymbolic","neurosymbolic-ai","policy-gradient","program-generator","pytorch","reinforce","reinforcement-learning","seq2seq","supervised-learning","system-2","transformer","university-project","visual-question-answering","visual-reasoning","vqa"],"created_at":"2025-11-11T11:16:58.995Z","updated_at":"2026-05-07T09:31:17.998Z","avatar_url":"https://github.com/msmrexe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Neurosymbolic VQA Program Generator\n\nThis project was developed as an exploration for the graduate-level **System 2 AI** course, which focuses on building models that move beyond the fast, intuitive pattern-matching of \"System 1\" (typical in many deep learning models) toward the slower, deliberate, multi-step reasoning characteristic of \"System 2\" cognition.\n\nAs such, this project implements a comprehensive Neurosymbolic framework based on the paper **\"Inferring and Executing Programs for Visual Reasoning\" (Johnson et al., 2017)**. The core idea is to translate ambiguous natural language questions (from the CLEVR dataset) into explicit, executable symbolic programs. This approach bridges the gap between neural perception and symbolic reasoning.\n\nTo build this neural-to-symbolic translator, we explore and compare three distinct learning paradigms:\n1.  **Supervised Learning** (with both LSTM and Transformer models)\n2.  **Reinforcement Learning** (using REINFORCE to fine-tune)\n3.  **In-Context Learning** (using a pre-trained LLM)\n\n**Note on Provenance:** This is an academic project. While the core model implementations, training/evaluation scripts, and overall repository structure are original, the foundational code for data preprocessing, program definitions, and the symbolic `ClevrExecutor` was adopted from the official paper repository: [facebookresearch/clevr-iep](https://github.com/facebookresearch/clevr-iep). This foundational code was then significantly refactored, modernized, and adapted to fit this project's modular design and personal requirements.\n\n## Features\n\n* **Multi-Paradigm Training**: Implements Supervised, REINFORCE, and In-Context Learning (ICL) strategies.\n* **Dual Architectures**: Provides both LSTM-based (with attention) and Transformer-based Seq2Seq models for the core generator.\n* **Symbolic Executor**: Includes a `ClevrExecutor` that runs the generated programs on scene data to produce concrete answers.\n* **Modular \u0026 Extensible**: Fully modular code for data preprocessing, model definitions, training loops, and evaluation.\n* **Detailed Guides**: A suite of Jupyter Notebooks provides a clean, step-by-step guide for running all experiments.\n\n## Core Concepts \u0026 Techniques\n\n* **Neurosymbolic AI**: Bridges the gap between connectionist (neural) models for perception and classic symbolic (logical) models for reasoning.\n* **Sequence-to-Sequence (Seq2Seq)**: The core framework used to translate question-sequences into program-sequences.\n* **Policy Gradients (REINFORCE)**: An RL algorithm used to fine-tune the model based on the *semantic correctness* (right answer) rather than just *syntactic correctness* (right program).\n* **In-Context Learning (ICL)**: A \"zero-training\" approach that leverages the pattern-recognition abilities of Large Language Models (LLMs) by providing examples in a prompt.\n* **Visual Question Answering (VQA)**: The target task, requiring the model to reason about visual scenes to answer natural language questions.\n\n---\n\n## How It Works\n\nThis project's goal is to solve a VQA task by generating a symbolic program that represents the reasoning steps needed to answer a question.\n\n### 1. The Neurosymbolic Approach: An Overview\n\nThe core idea is to decouple the problem:\n\n1.  **Neural Perception/Translation**: A Seq2Seq model (the \"neural\" part) reads the ambiguous natural language question and translates it into a structured, logical program.\n    * **Question**: \"Is there a small rubber cube behind the green cylinder?\"\n    * **Program**: `scene` $\\rightarrow$ `filter_shape[cylinder]` $\\rightarrow$ `filter_color[green]` $\\rightarrow$ `unique` $\\rightarrow$ `relate[behind]` $\\rightarrow$ `filter_shape[cube]` $\\rightarrow$ `filter_material[rubber]` $\\rightarrow$ `filter_size[small]` $\\rightarrow$ `exist`\n2.  **Symbolic Reasoning/Execution**: A symbolic executor (the \"symbolic\" part) takes this unambiguous program and executes it step-by-step against a structured representation of the scene to get the final answer.\n    * `scene` $\\rightarrow$ (gets all objects)\n    * `filter_shape[cylinder]` $\\rightarrow$ (keeps only cylinders)\n    * ...\n    * `exist` $\\rightarrow$ (checks if the final set of objects is empty) $\\rightarrow$ **\"yes\"**\n\nThis project focuses entirely on **Step 1**: building and training the best possible program generator.\n\n### 2. Data \u0026 Program Representation\n\n* **Input Data**: We use the **CLEVR dataset**, which consists of images (scenes) containing simple shapes (cubes, spheres, cylinders) of different sizes, colors, and materials.\n* **Questions**: Natural language questions about the scenes (e.g., \"How many...\", \"Are there...\").\n* **Programs**: The ground-truth data provides a \"functional program\" for each question. We preprocess these programs into a single string (in prefix notation by default) which becomes the target sequence for our models.\n    * e.g., `\u003cSTART\u003e exist filter_size[small] filter_material[rubber] ... \u003cEND\u003e`\n\n### 3. Implemented Learning Strategies\n\nWe implement and compare three different ways to train the program generator.\n\n#### A. Supervised Learning (Seq2Seq)\n\nThis is the baseline \"behavioral cloning\" approach. The model is trained to minimize the cross-entropy loss between its prediction and the ground-truth program, one token at a time.\n\n* **Objective**: Maximize the probability of the ground-truth program $Y$ given the question $X$.\n* **Loss Function**: Standard Cross-Entropy Loss (Teacher Forcing).\n\n  $$L_{\\text{SL}} = - \\sum_{t=1}^{T} \\log p(y_t | y_{\u003ct}, \\mathbf{X}; \\theta)$$\n  \n* **Problem**: This leads to **exposure bias**. The model is only trained on perfect, ground-truth prefixes. At inference, if it makes a single mistake, it may enter a state it has never seen, causing errors to cascade.\n\n#### B. Reinforcement Learning (REINFORCE)\n\nThis approach fine-tunes the supervised model to solve the exposure bias problem. Instead of forcing the model to match a specific program, we reward it for producing *any* program that gets the right answer.\n\n* **Policy ($\\pi_{\\theta}$)**: Our Seq2Seq model.\n* **Action ($a$)**: A full program *sampled* from the model's output distribution.\n* **Reward ($R$)**: We run the sampled program $a$ through the `ClevrExecutor`.\n    * $R = 1.0$ if the program's answer matches the ground-truth answer.\n    * $R = 0.0$ otherwise.\n* **Baseline ($b$)**: To stabilize training, we use a moving average of past rewards. The **Advantage** is $A = (R - b)$.\n* **Objective**: We use the REINFORCE algorithm to update the model's weights ($\\theta$) to maximize the expected reward. The loss is the negative policy gradient objective:\n\n  $$L_{\\text{RL}} = - \\mathbb{E}\\_{a \\sim \\pi_{\\theta}} [ (R - b) \\sum_{t=1}^{T} \\log \\pi_{\\theta}(a_t | a_{\u003ct}, \\mathbf{X}) ]$$\n\n  This \"pushes up\" the probability of programs that lead to a positive advantage and \"pushes down\" the probability of those that lead to a negative one.\n\n#### C. In-Context Learning (ICL)\n\nThis modern approach uses a large, pre-trained LLM and requires **no training or fine-tuning**. We leverage the LLM's powerful pattern-matching abilities by \"showing\" it examples of the task in its prompt.\n\n* **Prompt**: We construct a prompt containing a system message and $k$ \"shots\" (examples).\n* **Zero-Shot ($k=0$)**: The prompt only contains the instructions.\n* **Few-Shot ($k\u003e0$)**: The prompt contains instructions *and* $k$ examples of `(Question, Program)` pairs.\n* **Evaluation**: We test the LLM's ability to generate a correct program for a new, unseen question. We evaluate its performance as $k$ increases to see how quickly it \"learns\" the task.\n\n---\n\n## Project Structure\n\n```\nneurosymbolic-vqa-program-generator/\n├── .gitignore                               # Ignores data, logs, models, and Python caches\n├── LICENSE                                  # MIT License\n├── README.md                                # This file\n├── requirements.txt                         # Python dependencies\n├── data/\n│   └── .gitkeep                             # Placeholder for data/ (CLEVR\\_Dataset/ goes here)\n├── logs/\n│   └── .gitkeep                             # Placeholder for logs/ (e.g., train.log)\n├── models/\n│   └── .gitkeep                             # Placeholder for models/ (e.g., lstm.pth)\n├── notebooks/\n│   ├── 0_Data_Exploration_and_Setup.ipynb   # Guide: Download \u0026 preprocess data\n│   ├── 1_Supervised_Training.ipynb          # Guide: Run supervised experiments\n│   ├── 2_Reinforce_Finetuning.ipynb         # Guide: Run RL experiments\n│   └── 3_ICL_Evaluation.ipynb               # Guide: Run LLM ICL experiments\n├── scripts/\n│   ├── preprocess_data.py                   # Runnable script to build vocab \u0026 H5 files\n│   ├── train.py                             # Main entry point for training (Supervised \u0026 RL)\n│   └── evaluate.py                          # Runnable script to evaluate a trained model\n└── src/\n├── __init__.py\n├── config.py                                # Stores all paths and hyperparameters\n├── data_loader.py                           # Contains ClevrQuestionDataset and DataLoader\n├── executor.py                              # Contains the ClevrExecutor for running programs\n├── vocabulary.py                            # Helper functions for building/loading vocabs\n├── models/\n│   ├── __init__.py\n│   ├── base_rnn.py                          # Base class for RNNs\n│   ├── lstm_seq2seq.py                      # LSTM Encoder, Decoder, and Attention\n│   └── transformer_seq2seq.py               # Transformer model implementation\n├── training/\n│   ├── __init__.py\n│   ├── train_reinforce.py                   # Trainer class for the REINFORCE loop\n│   └── train_supervised.py                  # Trainer class for the Supervised loop\n├── evaluation/\n│   ├── __init__.py\n│   ├── eval_icl.py                          # Logic for running ICL evaluation\n│   └── eval_model.py                        # Logic for evaluating our trained models\n└── utils/\n    ├── __init__.py\n    ├── logger.py                            # Sets up file and console logging\n    ├── program_utils.py                     # Helpers for program list/tree/string conversions\n    └── scene_utils.py                       # Helper for loading scene JSONs\n```\n\n## How to Use\n\n1.  **Clone the Repository:**\n    ```bash\n    git clone https://github.com/msmrexe/neurosymbolic-vqa-program-generator.git\n    cd neurosymbolic-vqa-program-generator\n    ```\n\n2.  **Setup Environment and Data:**\n    ```bash\n    # Install dependencies\n    pip install -r requirements.txt\n    \n    # Download the CLEVR dataset (see Notebook 0 for link)\n    # Unzip and place it in the `data/` folder.\n    # The path should be: data/CLEVR_Dataset/\n    ```\n\n3.  **Run Preprocessing:**\n    First, preprocess the `train` data to create the vocabulary. Then, use that vocabulary to preprocess the `val` and `test` data.\n\n    ```bash\n    # Process TRAIN (creates vocab)\n    python scripts/preprocess_data.py \\\n        --input_json data/CLEVR_Dataset/Questions/CLEVR_train_questions.json \\\n        --output_h5 data/dataH5Files/clevr_train_questions.h5 \\\n        --output_vocab_json data/dataH5Files/clevr_vocab.json\n\n    # Process VAL (uses existing vocab)\n    python scripts/preprocess_data.py \\\n        --input_json data/CLEVR_Dataset/Questions/CLEVR_val_questions.json \\\n        --input_vocab_json data/dataH5Files/clevr_vocab.json \\\n        --output_h5 data/dataH5Files/clevr_val_questions.h5 \\\n        --allow_unk 1\n    ```\n\n4.  **Run Experiments (via Notebooks):**\n    The easiest way to run the project is to follow the Jupyter Notebooks in the `notebooks/` directory.\n\n    ```bash\n    jupyter lab notebooks/\n    ```\n    * **`0_Data_Exploration_and_Setup.ipynb`**: Confirms data is set up correctly.\n    * **`1_Supervised_Training.ipynb`**: Runs supervised training for both models.\n    * **`2_Reinforce_Finetuning.ipynb`**: Runs REINFORCE fine-tuning.\n    * **`3_ICL_Evaluation.ipynb`**: Runs the final LLM-based evaluation.\n\n5.  **Run Experiments (via Terminal):**\n    You can also run the scripts directly from the command line.\n\n    ```bash\n    # Example: Supervised Training (LSTM)\n    python scripts/train.py \\\n        --model_type lstm \\\n        --train_mode supervised \\\n        --model_save_path models/supervised_lstm.pth \\\n        --num_iters 100000\n\n    # Example: REINFORCE Fine-Tuning (LSTM)\n    python scripts/train.py \\\n        --model_type lstm \\\n        --train_mode reinforce \\\n        --load_model models/supervised_lstm.pth \\\n        --model_save_path models/reinforce_lstm.pth \\\n        --num_iters 50000 \\\n        --learning_rate 1e-5\n\n    # Example: Evaluate a model\n    python scripts/evaluate.py \\\n        --model_type lstm \\\n        --model_path models/reinforce_lstm.pth \\\n        --data_h5_path data/dataH5Files/clevr_val_questions.h5 \\\n        --split val\n    ```\n\n---\n\n## Author\n\nFeel free to connect or reach out if you have any questions!\n\n* **Maryam Rezaee**\n* **GitHub:** [@msmrexe](https://github.com/msmrexe)\n* **Email:** [ms.maryamrezaee@gmail.com](mailto:ms.maryamrezaee@gmail.com)\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for full details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsmrexe%2Fneurosymbolic-vqa-program-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmsmrexe%2Fneurosymbolic-vqa-program-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmsmrexe%2Fneurosymbolic-vqa-program-generator/lists"}