{"id":21314630,"url":"https://github.com/dor-sketch/irisnetrbm","last_synced_at":"2025-03-15T21:22:41.975Z","repository":{"id":225707469,"uuid":"764898811","full_name":"Dor-sketch/IrisNetRBM","owner":"Dor-sketch","description":"A Python-implemented RBM project exploring generative learning through the classification of the Iris dataset, featuring a user-friendly GUI and advanced data handling capabilities.","archived":false,"fork":false,"pushed_at":"2024-03-12T20:03:12.000Z","size":1810,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-22T10:23:03.267Z","etag":null,"topics":["iris-dataset","machine-learning","neural-network","rbm","restricted-boltzmann-machine"],"latest_commit_sha":null,"homepage":"https://en.wikipedia.org/wiki/Iris_flower_data_set","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Dor-sketch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-28T22:57:09.000Z","updated_at":"2024-06-17T19:44:18.000Z","dependencies_parsed_at":"2024-03-03T21:24:16.508Z","dependency_job_id":"9ca49e61-1dbb-4e28-b66e-28d7572ed25c","html_url":"https://github.com/Dor-sketch/IrisNetRBM","commit_stats":null,"previous_names":["dor-sketch/irisnetrbm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dor-sketch%2FIrisNetRBM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dor-sketch%2FIrisNetRBM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dor-sketch%2FIrisNetRBM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Dor-sketch%2FIrisNetRBM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Dor-sketch","download_url":"https://codeload.github.com/Dor-sketch/IrisNetRBM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243791136,"owners_count":20348423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["iris-dataset","machine-learning","neural-network","rbm","restricted-boltzmann-machine"],"created_at":"2024-11-21T18:14:19.388Z","updated_at":"2025-03-15T21:22:41.941Z","avatar_url":"https://github.com/Dor-sketch.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Restricted Boltzmann Machine (RBM) 🧠\n\nWelcome to my **Restricted Boltzmann Machine (RBM)** project, a key part of the **CA 20581 Biological Computation** course, 2024a, at *the Open University of Israel*. This project delves into the generative stochastic neural networks, aiming to learn complex probability distributions from input data.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/RBM_cover.png\" alt=\"RBM Cover\" width=\"400\"\u003e\n\u003c/p\u003e\n\n---\n\n\u003c!-- @import \"[TOC]\" {cmd=\"toc\" depthFrom=2 depthTo=3 orderedList=false} --\u003e\n\n\u003c!-- code_chunk_output --\u003e\n\n- [Project Overview 📜](#project-overview-)\n  - [Background](#background)\n  - [Features](#features)\n- [Getting Started 🚀](#getting-started-)\n  - [Command-Line Interface ⌨️](#command-line-interface-️)\n  - [GUI Interface 🖥️](#gui-interface-️)\n- [RBM Core Architecture 🏗️](#rbm-core-architecture-️)\n  - [Component Overview](#component-overview)\n  - [Network Design](#network-design)\n  - [Layer Classes and Hierarchy](#layer-classes-and-hierarchy)\n  - [Network Structure Illustration](#network-structure-illustration)\n  - [Important Note](#important-note)\n  - [Energy function](#energy-function)\n  - [Initialization Process](#initialization-process)\n- [Utilizing the Network](#utilizing-the-network)\n  - [Evaluating the Model](#evaluating-the-model)\n  - [Initial Observations](#initial-observations)\n  - [Understanding the Outcomes](#understanding-the-outcomes)\n- [Contributing to the Project 💡](#contributing-to-the-project-)\n\n\u003c!-- /code_chunk_output --\u003e\n\n---\n\n## Project Overview 📜\n\n### Background\n\nRestricted Boltzmann Machines (RBMs) are generative stochastic neural networks adept at learning probability distributions over input sets. Featuring both visible and hidden units with undirected connections, RBMs harness the contrastive divergence algorithm for efficient parameter optimization.\n\nThis RBM implementation stems from a Biological Computation course at the Open University of Israel, designed to delve into generative learning's nuances and RBMs' application in machine learning and AI. Remarkably, this project foregoes external libraries, utilizing only Python and NumPy to construct the RBM and its components.\n\nThe project leverages the Iris dataset, a staple in machine learning for its detailed feature set of iris flower species, to train the RBM in classifying iris flowers, showcasing the model's prowess in learning complex distributions and executing classification tasks. Additionally, a GUI application enhances user engagement with the RBM, offering an intuitive interface for dataset interaction.\n\n### Features\n\nThis RBM project encompasses a suite of features to navigate the intricacies of generative learning and neural networks:\n\n- **Customizable Hidden Units:** Adapt the RBM with varying hidden unit counts to accommodate different datasets and learning challenges.\n- **Efficient Learning with Contrastive Divergence:** Employ this cutting-edge algorithm for streamlined learning and model refinement.\n- **Robust Model State Management:** Effortlessly manage the model's weights, biases, and settings, facilitating continuity in projects or analyses.\n- **Insightful Visualizations:** Gain a deeper understanding of the model through comprehensive visualization tools that illustrate the network's structure, weights, biases, and learning evolution.\n- **Interactive User Experience:** Engage with the RBM via a polished GUI or command-line interface, designed for ease of data management, model training, and evaluation.\n- **Advanced Data Preprocessing:** Utilize the `DataSet` class for proficient dataset management, ensuring smooth integration with the RBM's generative learning requirements.\n\n---\n\n## Getting Started 🚀\n\nInteracting with the RBM model is streamlined through both a command-line interface and a GUI application, offering varied functionalities for model training, evaluation, and visualization.\n\n### Command-Line Interface ⌨️\n\nNavigate the RBM model with these command-line flags for a tailored interaction:\n\n- `-train`: Initiates model training with designated or default settings.\n- `-test`: Evaluates the model's accuracy on a given dataset.\n- `-plot`: Visualizes the model's structure, weights, and biases.\n- `-save`: Archives the model's weights and biases for future use.\n- `-load`: Restores the model's weights and biases from saved files.\n\n#### Example Command 📌\n\nTo train the model, and then save the weights:\n\n```bash\npython rbm.py -train -save\n```\n\nTo load an existing model and test its performance:\n\n```bash\npython rbm.py -load -test\n```\n\n### GUI Interface 🖥️\n\nTo enhance user interaction and simplify the process of managing the RBM, this project includes a GUI application built with Tkinter. The GUI provides an intuitive interface for performing key operations such as data loading, model training/testing, and visualization.\n\nFrom the GUI, you can load the dataset, train the network, test the network, and plot the synapses and biases of the network. I tried to make it friendly and easy to use. Make sure to load the dataset before training or testing the network, a file dialog will open when you click the `Load` button. The [iris.data](iris.data) file is included in the repository.\n\n![alt text](./images/GUI.png)\n\n---\n\n## RBM Core Architecture 🏗️\n\nThis section outlines the core components of the Restricted Boltzmann Machine (RBM) network, implemented in Python. The architecture is modular, designed for flexibility across various datasets, with specific optimization for the Iris dataset.\n\n### Component Overview\n\n- **[DataSet.py](DataSet.py)**: Defines the `DataSet` class, responsible for loading and parsing datasets into numpy arrays of discrete values, tailored for RBM processing.\n- **[RBM.py](RBM.py)**: Introduces the `RBM` class, embodying the core functionality of the RBM network, including training and sampling methods. This file also contains the `Synapse` class, which is currently under review for removal due to potential redundancy.\n- **[GUI.py](GUI.py)**: Implements a graphical user interface (`GUI` class) to facilitate user interaction with the RBM network.\n- **[layers.py](layers.py)**: Contains the `Layer` class, representing individual layers within the RBM. This class is crucial for managing neuron states, biases, and potentially enabling parallel processing within the network. The structure aims to highlight network restrictions and enhance modularity.\n\n### Network Design\n\nThe RBM is configured for the Iris dataset as follows:\n\n- **Input Neurons**: 12 neurons, allocated as 3 per each of the 4 dataset features, post-normalization to discrete values.\n- **Output Neurons**: 3 neurons, one per dataset class.\n- **Hidden Neurons**: 16 neurons, an empirically determined number based on experimental outcomes.\n\nNote that the input and output neuron are the `visible units` ( $\\vec{v} = (v_1,...,v_{n=15})$ ) and the `hidden units` are the hidden neurons ( $\\vec{h} = (h_1,...,h_{m=16})$ ).\n\n### Layer Classes and Hierarchy\n\nThe network employs two primary layer types, managed through inheritance:\n\n- **`Layer`**: A generic class for RBM layers, managing units (neurons) and biases. It serves as the foundation for:\n  - **`HiddenLayer`** and **`VisibleLayer`**: Specialized classes derived from `Layer`, tailored to their specific roles in the network structure.\n\n### Network Structure Illustration\n\n![RBM Network Structure](images/bfs_network.png)\n\nThe diagram details the network's composition, including hidden neurons ($h_0$ to $h_{15}$), input neurons ($v_0$ to $v_{11}$), and output neurons ($v_{12}$ to $y_{14}$), with 0-based indexing for clarity and consistency.\n\n### Important Note\n\nAn indexing discrepancy led to a significant bug affecting class classification accuracy. The issue, stemming from a mix of 1-based and 0-based indexing, primarily impacted the third class's recognition. This has been rectified to ensure consistent and accurate network performance.\n\n### Energy function\n\n```python\ndef energy(self, visible_units, hidden_units):\n    \"\"\"\n    calculate the energy of the model using the formula:\n    -Σ(vi * a) - Σ(hj * b) - Σ(Σ(vi * wji) * hj)\n    args:\n    - visible_units: the visible layer units\n    - hidden_units: the hidden layer units\n    \"\"\"\n    visible_bias = self.visible_layer.bias\n    hidden_bias = self.hidden_layer.bias\n    weights = self.synapses.weights\n    return -np.dot(visible_bias, visible_units)\\\n        - np.dot(hidden_bias, hidden_units) - \\\n        np.dot(visible_units.T @ weights,\n               hidden_units)  # note the @ operator\n```\n\nThe energy function of the network is thus:\n\n$E(\\vec{v}, \\vec{h}) = -\\sum_{i=1}^{15} v_i a_i - \\sum_{j=1}^{16} h_j b_j - \\sum_{i=1}^{15} \\sum_{j=1}^{16} J_{ij} v_i h_j$\n\nWhere:\n\n### Initialization Process\n\n#### Data Handling with DataSet Class\n\nThe `DataSet` class serves as the cornerstone for data management within the RBM framework. It is engineered to streamline the loading, preprocessing, and handling of datasets, ensuring seamless compatibility with the RBM's operational requirements.\n\nKey Features:\n\n- **Instance Management**: Facilitates the encapsulation of data entries through the `Instance` object, streamlining data manipulation and access.\n- **Discrete Value Transformation**: Employs a method to convert continuous attributes into discrete values, optimizing data for the RBM's generative learning process.\n- **Adaptable Data Processing**: Supports dynamic integration of labels, attributes, and instances, catering to diverse dataset structures.\n- **Efficient Data Loading**: Implements the `createDataSet` method for direct file-based data initialization, accommodating real values that are discretized as necessary.\n\n##### Operational Workflow\n\n1. **Setup**: Initiates with an empty framework for labels, attributes, and instances.\n2. **Data Ingestion**: Utilizes file input to populate the model with relevant data, identifying and storing components via specialized methods.\n3. **Data Conversion**: Applies the `convertToDiscrete` method to continuous values, preparing the dataset for RBM training.\n\n#### Synaptic Initialization\n\nBefore runing the classification the following steps must preceed:\n\n1. Define the energy function with suitable parameters.\n2. Lock the input units as required.\n3. Initialize the other units with random values, and choose big $T$ to allow the network to converge.\n\nThe synapses and biases are initialized using the `train` method, which is called from the `RBM` class. The process is as follows:\n\n1. **Synaptic Initialization**: The weights and biases are initialized using a random number generator from the numpy library, with values ranging between -0.1 and 0.1, laying the foundation for diverse neural connections.\n2. **Choose the learning rate**: The learning rate is set to 0.1, a value that has been empirically determined to facilitate efficient learning and convergence.\n\nThan the following steps are reapeated until the network converges, or until a maximum number of iterations is reached:\n\n1. Pick a random instance $\\vec{v} = (v_1,...,v_15)$ from the dataset.\n\n2. Calculate the probability $P_k$ for every hidden unit $h_k$ using the formula: $P_k = \\frac{1}{1 + e^{-\\sum_{i=1}^{15} v_i w_{ik} - b_k}}$. Note that that it doesnt deapend on the other hidden units - this is the reason for the name `Restricted` Boltzmann Machine, and also why it enables parallel processing.\n\n3. Initialize the classification algorithm with $\\vec{v} = (v_1,...,v_{15})$ and perform a single step of the Gibbs sampling algorithm to get network state $(\\vec{h}(1), \\vec{v}(1))$. This is done without locking the input units.\n\n4. Update the weights and biases using the contrastive divergence algorithm:\n   - $a_i^{new} = a_i + \\eta(v_i - v_i{(1)})$\n   - $b_j^{new} = b_j + \\eta(P_j - h_j{(1)})$\n   - $J_{ij}^{new} = J_{ij} + \\eta(v_i P_j - v_i{(1)} h_j{(1)})$\n\n5. Repeat the previous steps until the network converges, or until a maximum number of iterations is reached.\n\n## Utilizing the Network\n\n### Evaluating the Model\n\n- **Pre-training Evaluation**: Utilize the GUI or modify `CONSOLE_LOGGING` in `RBM.py` to `True` for insights into the network's initial state.\n- **Training Execution**: Initiate the learning process via GUI or command line, employing the `test` function.\n\n### Initial Observations\n\nPost-training, the network exhibits around 90% accuracy on the test set, demonstrating stable performance and precise classification across multiple iterations. For custom runs, leverage the `load` argument or the GUI to input predefined biases and weights.\n\n### Understanding the Outcomes\n\nThe evolution of biases and synapses post-learning is visually represented, indicating convergence towards optimal values and illustrating the learning impact.\n\n| Condition | Before Learning | After Learning |\n|-----------|-----------------|----------------|\n| Biases    | ![Before](images/untrained_biases.png) | ![After](images/trained_biases.png) |\n| Synapses  | ![Before](images/untrained_weights.png) | ![After](images/trained_weights.png) |\n\n## Contributing to the Project 💡\n\nYour expertise can significantly propel this project forward. We invite contributions in all forms, from code enhancements to feedback. Join us in refining and expanding the capabilities of this RBM implementation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdor-sketch%2Firisnetrbm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdor-sketch%2Firisnetrbm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdor-sketch%2Firisnetrbm/lists"}