{"id":15176663,"url":"https://github.com/matteofasulo/transformers-sentence-reconstruction","last_synced_at":"2026-01-06T00:49:42.712Z","repository":{"id":249573919,"uuid":"831870930","full_name":"MatteoFasulo/Transformers-Sentence-Reconstruction","owner":"MatteoFasulo","description":"Sentence Reconstruction using Transformer Model","archived":false,"fork":false,"pushed_at":"2024-07-21T21:36:44.000Z","size":65,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T16:15:03.960Z","etag":null,"topics":["keras","keras-nlp","tensorflow","transformers"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MatteoFasulo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-21T21:17:50.000Z","updated_at":"2024-07-21T21:37:19.000Z","dependencies_parsed_at":"2024-07-21T23:49:45.757Z","dependency_job_id":null,"html_url":"https://github.com/MatteoFasulo/Transformers-Sentence-Reconstruction","commit_stats":null,"previous_names":["matteofasulo/transformers-sentence-reconstruction"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatteoFasulo%2FTransformers-Sentence-Reconstruction","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatteoFasulo%2FTransformers-Sentence-Reconstruction/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatteoFasulo%2FTransformers-Sentence-Reconstruction/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MatteoFasulo%2FTransformers-Sentence-Reconstruction/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MatteoFasulo","download_url":"https://codeload.github.com/MatteoFasulo/Transformers-Sentence-Reconstruction/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245689514,"owners_count":20656417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["keras","keras-nlp","tensorflow","transformers"],"created_at":"2024-09-27T13:23:19.453Z","updated_at":"2026-01-06T00:49:42.660Z","avatar_url":"https://github.com/MatteoFasulo.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📚 Sentence Reconstruction using Transformer Model\n\n## 🚀 Introduction\n\nThis project is a comprehensive exploration of the application of deep learning techniques to the problem of sentence reconstruction. The task is to reconstruct the original sequence of a given English sentence from a random permutation of its words. This is a challenging problem due to the inherent complexity of language and the numerous possible permutations of a sentence.\n\n## 🌟 Project Overview\n\nThe project is constrained by several factors:\n\n* ❌ **No use of pretrained models:** The model must learn the task from scratch, without the benefit of prior knowledge encoded in models like BERT or GPT.\n* 📏 **Model size limitation:** The neural network model size is limited to less than 20M parameters, ensuring efficiency and the ability to train and deploy on modest hardware.\n* 🚫 **No postprocessing techniques:** The model’s output must be used directly, without additional processing to select the most likely sequence.\n* 📊 **Limited training data:** The model must learn from the provided dataset alone.\n\n## 🛠️ Libraries and Frameworks\n\nThe project is implemented in Python, using:\n\n- **Keras** for building and training the neural network model.\n- **TensorFlow** as the backend for Keras, providing efficient, GPU-accelerated computations.\n- **Hugging Face** library for providing the datasets for training and testing the model.\n\n## ⚙️ Preprocessing and Model Design\n\nThe project includes extensive preprocessing of the dataset. This involves tokenizing the sentences, encoding the words as integers, and creating the random permutations. A custom text vectorization layer is created for this purpose.\n\nThe model itself is a Transformer, a type of neural network that uses self-attention mechanisms to capture the dependencies between words in a sentence. The model has an encoder-decoder architecture, where the encoder processes the input sentence and the decoder generates the reconstructed sentence.\n\n## 🏗️ Model Architecture\n\nThe Transformer model architecture used in this project includes:\n\n- **Embedding Layer**: Converts input tokens into dense vectors of fixed size.\n- **Encoder**: Multiple layers of self-attention and feed-forward neural networks.\n- **Decoder**: Similar to the encoder but designed to produce the output sequence.\n- **Positional Encoding**: Adds information about the position of each token in the sequence.\n\n## 🏋️ Training\n\nThe training process involves the following steps:\n\n1. **Data Preparation**: Tokenize and encode the input sentences and their shuffled versions.\n2. **Model Initialization**: Define the Transformer model with specified hyperparameters.\n3. **Loss Function and Optimizer**: Use cross-entropy loss and Adam optimizer.\n4. **Training Loop**: Train the model over several epochs, adjusting weights to minimize the loss.\n\n## 📊 Evaluation\n\nThe model is evaluated using a specific metric that measures the accuracy of the reconstructed sentences. This metric finds the longest common subsequence between the original and reconstructed sentences and calculates the ratio of this length to the length of the original sentence. A higher ratio indicates a better reconstruction. The model is tested on a separate test set to assess its performance. The evaluation process involves:\n\n- Generating reconstructed sentences from the shuffled input.\n- Comparing the reconstructed sentences with the original sentences.\n\n## 🏆 Results\n\nThe proposed Transformer model demonstrated good performance in reconstructing sentences. It outperformed Seq2Seq models with LSTM encoders and decoders in capturing long-term dependencies and syntactic structures.\n\n### 📈 Performance Metrics and Testing\n\nThe Transformer model is tested on a set of 3,000 randomly selected instances. The results are promising, with the model achieving an average score of approximately 0.51 with a standard deviation of 0.28. This indicates that the model is able to reconstruct the original sentence with reasonable accuracy.\n\nThe model’s architecture, which has less than 10 million parameters, is discussed in detail. The impact of various hyperparameters, such as the number of layers and the embedding dimensions, on the model’s size and performance is analyzed.\n\n## 🔍 Conclusions and Future Work\n\nThe Transformer model shows promise in the task of sentence reconstruction. It is able to capture long-term dependencies and syntactic structures in the sentences, outperforming previous LSTM-based Seq2Seq models. This is achieved within the constraints of the project’s parameter limit.\n\n### 💡 Potential Improvements\n\n- **Model Optimization**: Increasing the number of layers and the embedding dimensions to further enhance performance.\n- **Custom Loss Function**: Implementing a custom loss function that penalizes shorter sequences could help improve the model's accuracy.\n- **Hyperparameter Tuning**: Exploring different hyperparameter configurations to achieve better results.\n- **Attention Visualization**: Visualizing attention weights to better understand how the model processes and reconstructs sentences.\n\nThe project concludes by selecting the model with fewer parameters as the more challenging and interesting solution. This model strikes a balance between performance and efficiency, and represents a promising direction for future work.\n\n## 📚 References\n\n- Vaswani, A., et al. \"Attention Is All You Need.\" *Advances in Neural Information Processing Systems*, 2017.\n- [Transformers by Hugging Face](https://huggingface.co/transformers/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatteofasulo%2Ftransformers-sentence-reconstruction","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmatteofasulo%2Ftransformers-sentence-reconstruction","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmatteofasulo%2Ftransformers-sentence-reconstruction/lists"}