{"id":30527986,"url":"https://github.com/proshian/neural-swipe-typing","last_synced_at":"2025-08-27T04:03:13.493Z","repository":{"id":207462724,"uuid":"708145025","full_name":"proshian/neural-swipe-typing","owner":"proshian","description":"A transformer that decodes swipes across a smartphone keyboard into words (gesture / swipe / glide typing) (enhanced yandex cup solution)","archived":false,"fork":false,"pushed_at":"2025-07-16T12:06:59.000Z","size":25611,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-17T12:00:13.186Z","etag":null,"topics":["gesture-keyboard","gesture-typing","glide-typing","keyboard","pytorch","shape-typing","shape-writing","swipe","swipe-typing","transformer","yandex-cup"],"latest_commit_sha":null,"homepage":"https://proshian.pythonanywhere.com/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proshian.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-10-21T16:53:07.000Z","updated_at":"2025-03-06T15:16:52.000Z","dependencies_parsed_at":"2024-05-15T12:26:12.187Z","dependency_job_id":"6141e78e-93f4-41b5-abae-672e5226a22f","html_url":"https://github.com/proshian/neural-swipe-typing","commit_stats":null,"previous_names":["proshian/yandex_cup_2023_ml_neuroswipe","proshian/yandex-cup-2023-ml-neuroswipe","proshian/neuroswipe","proshian/neural-glide-typing","proshian/neural-swipe-typing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/proshian/neural-swipe-typing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proshian%2Fneural-swipe-typing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proshian%2Fneural-swipe-typing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proshian%2Fneural-swipe-typing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proshian%2Fneural-swipe-typing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proshian","download_url":"https://codeload.github.com/proshian/neural-swipe-typing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proshian%2Fneural-swipe-typing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272288931,"owners_count":24907776,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-27T02:00:09.397Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gesture-keyboard","gesture-typing","glide-typing","keyboard","pytorch","shape-typing","shape-writing","swipe","swipe-typing","transformer","yandex-cup"],"created_at":"2025-08-27T04:01:55.600Z","updated_at":"2025-08-27T04:03:13.376Z","avatar_url":"https://github.com/proshian.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# Neural glide typing\n\nA transformer neural network for a gesture keyboard that transduces curves swiped across a keyboard into word candidates\n\nContribution:\n* **A new method for constructing swipe point embeddings (SPE) that outperforms existing ones.** It leverages a weighted sum of all keyboard key embeddings, resulting in a notable perfomance boost: **0.67% increase in Swipe MRR and 0.73% in accuracy** compared to SPE construction methods described in literature\n\nOther highlights:\n* **Enhanced Inference with Custom Beam Search**: a modified beam search is implemented that masks out logits corresponding to impossible (according to dictionary) token continuations given an already generated prefix. It is faster and more accurate than a standard beam search\n\n*This repository used to contain my Yandex Cup 2023 solution (7th place), but after many improvements, it has become a standalone project*\n\n## Demo\n\nTry out a live demo with a trained model from the competition through this [web app](https://proshian.pythonanywhere.com/)\n\n\n![demo](./docs_and_assets/swipe_demos/demo.gif)\n\n\u003e [!Note]\n\u003e If the website is not available, you can run the demo yourself by following the instructions in [the web app's GitHub repository](https://github.com/proshian/neuroswipe_inference_web).\n\n\u003e [!Note]\n\u003e The website may take a minute to load, as it is not yet fully optimized. If you encounter a \"Something went wrong\" page, try refreshing the page. This usually resolves the issue.\n\n\u003e [!NOTE]  \n\u003e The model is an old and underfit legacy transformer variation (m1_bigger in models.py) that was used in the competition. A significant update is planned for both this project and the web app, but it will happen in winter 2024 \n\n## Report\n\n**Access a brief research report [here](docs_and_assets/report/report.md)**, which includes:\n\n* Overview of existing research\n* Description of the developed method for constructing swipe point embeddings\n* Comparative analysis and results\n\nFor in-depth insights, you can refer to my [master's thesis](https://drive.google.com/file/d/1ad9zlfgfy6kOA-41GxjUQIzr8cWuaqxL/view?usp=sharing) (in Russian)\n\n\n## Prerequisites\n\nInstall the dependencies:\n\n```sh\npip install -r requirements.txt\n```\n\n* The inference was tested with python 3.10\n* The training was conducted in kaggle using Tesla P100\n\n\n\n## Yandex Cup Dataset: Obtaining and Preparation\n\nTo acquire and prepare the Yandex Cup dataset, follow the steps below:\n\n### Option 1: Obtain and Preprocess the Dataset from Scratch\n\n```sh\ncd src\nbash ./data_obtaining_and_preprocessing/obtain_and_prepare_data.sh\n```\n\n\u003e [!Note]  \n\u003e The pipeline takes approximately **6 hours** to complete on the tested machine.\n\n\n### Option 2: Download the Preprocessed Dataset (Recommended)\n\nIf you prefer to skip the lengthy preprocessing steps, you can directly download the preprocessed dataset:\n\n```sh\ncd src\npython ./data_obtaining_and_preprocessing/download_dataset_preprocessed.py\n```\n\n\n\n## Workflow Overview\n\nA trained model is defined not only by its class and weights but also by the dataset transformation used during training.\n\n\nAll current models are instances of `model.EncoderDecoderTransformerLike` and consist of the following components:\n* Swipe point embedder\n* Word component token embedder (currently char-level)\n* Encoder\n* Decoder \n\nTransforms extract features from the raw dataset, converting each dataset item from the format `(x, y, t, grid_name, tgt_word)` to `(encoder_input, decoder_input), decoder_output`.\n\nAfter collating the dataset, the format becomes `(packed_model_in, dec_out)`, where `packed_model_in` is `(encoder_input, decoder_input, swipe_pad_mask, word_pad_mask)`. `packed_model_in` is passed to the model via unpacking (`model(*packed_model_in)`).\n\n* `encoder_input` is passed as the only argument to swipe_point_embedder’s forward. The type depends on which swipe point embedding layer you use. It can be a single object, a tuple of objects\n* `decoder_input` and `decoder_output` are `tokenized_target_word[1:]` and `tokenized_target_word[:-1]` correspondingly.\n\n\nA trained swipe decoding method is defined by\n* model class\n* model weights\n* dataset transformation\n* decoding algorithm\n\n\n\n## Your Custom Dataset\n\nYour custom dataset must have items of format: `tuple(x, y, t, grid_name, tgt_word)`. These raw features won't be used but there are transforms defined in `feature_extractors.py` corresponding to every type of `swipe point embedding layer` that extract the needed features. You can apply these transforms in your dataset's `__init__` method or in `__get_item__` / `__iter__`. The data formats after transform and after collation are described above\n\nYou also need to add your keyboard layout to `grid_name_to_grid.json`\n\n\u003c!--\n\n**TODO: Add info on how exactly the dataset should be integrated** \n\n--\u003e\n\n## Training\n\n\u003c!-- Перед побучением необходимо очистить тренировочный датасет --\u003e\n\nThe training is done in [train.ipynb](src/train.ipynb)\n\n\u003e [!WARNING]  \n\u003e `train.ipynb` drains RAM when using `n_workers` \u003e 0 in Dataloader. This can result in up to `dataset_size * n_workers` extra gigabytes of RAM usage. This is a known issue (see [here](https://github.com/pytorch/pytorch/issues/13246)) that happens when a dataset uses a list to store data. Although `torch.cuda.empty_cache()` can be used as a workaround, it doesn't seem to work with pytorch lightning. It appears I didn't commit this workaround, but you can adapt train.ipynb from [before-lightning branch](https://github.com/proshian/neuroswipe/tree/before-lightning) by adding ```torch.cuda.empty_cache()``` after each epoch to to avoid the issue. When training in a kaggle notebook, the issue is not a problem since a kaggle session comes with 30 Gb of RAM.  \n\n\n## Prediction\n\n[word_generation_demo.ipynb](src/word_generation_demo.ipynb) serves as an example on how to predict via a trained model.\n\n[predict_v2.py](src/predict_v2.py) is used to obtain word candidates for a whole dataset and pickle them\n\npredict_v2.py usage example:\n\n```\npython3.10 src/predict_v2.py --config configs/config__my_weighted_features.json --num-workers 0\n```\n\n\u003e [!WARNING]  \n\u003e If the decoding algorithm in `predict_v2.py` script utilizes a vocabulary for masking (if `use_vocab_for_generation: true` in the config), it is necessary to disable multiprocessing by passing the command-line argument `--num-workers 0` to the script. Otherwise, the prediction will take a long time. It's a bug that will be fixed\n\n\n\n## Yandex cup 2023 results\n* [task](./docs_and_assets/yandex_cup/task/task.md)\n* [submission reproduction](./docs_and_assets/yandex_cup/submission_reproduciton_instrucitons.md). \n* [leaderboard](./docs_and_assets/yandex_cup/leaderboard.md)\n\n\n# Documentation\nA WIP documentation can be found [here](./docs_and_assets/documentation.md). It doesn't contain much information yet, will be updated. Please refer to docstrings in the code for now\n\n\n## Thank you for your attention\n![thank_you](./docs_and_assets/swipe_demos/thank_you.gif)\n\n## For future me\nSee [refactoring plan](./docs_and_assets/Refactoring_plan.md)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproshian%2Fneural-swipe-typing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproshian%2Fneural-swipe-typing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproshian%2Fneural-swipe-typing/lists"}