{"id":13698768,"url":"https://github.com/bryanlimy/tf2-transformer-chatbot","last_synced_at":"2026-03-15T13:46:52.256Z","repository":{"id":37795684,"uuid":"181243075","full_name":"bryanlimy/tf2-transformer-chatbot","owner":"bryanlimy","description":"Transformer Chatbot in TensorFlow 2 with TPU support.","archived":false,"fork":false,"pushed_at":"2022-08-06T16:00:48.000Z","size":1611,"stargazers_count":130,"open_issues_count":8,"forks_count":59,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-07T23:36:29.757Z","etag":null,"topics":["chatbot","deep-learning","tensorflow","tensorflow-2","tensorflow-tutorials","tensorflow2","transformer"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bryanlimy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-14T00:34:16.000Z","updated_at":"2025-03-28T17:15:30.000Z","dependencies_parsed_at":"2022-06-23T17:03:40.017Z","dependency_job_id":null,"html_url":"https://github.com/bryanlimy/tf2-transformer-chatbot","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bryanlimy/tf2-transformer-chatbot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2Ftf2-transformer-chatbot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2Ftf2-transformer-chatbot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2Ftf2-transformer-chatbot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2Ftf2-transformer-chatbot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bryanlimy","download_url":"https://codeload.github.com/bryanlimy/tf2-transformer-chatbot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanlimy%2Ftf2-transformer-chatbot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271409427,"owners_count":24754712,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","deep-learning","tensorflow","tensorflow-2","tensorflow-tutorials","tensorflow2","transformer"],"created_at":"2024-08-02T19:00:52.691Z","updated_at":"2026-03-15T13:46:52.219Z","avatar_url":"https://github.com/bryanlimy.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook","Projects 💛💛💛💛💛\u003ca name=\"Projects\" /\u003e"],"sub_categories":["NLP"],"readme":"## Transformer Chatbot with TensorFlow 2\r\nBuild an end-to-end chatbot with [Transformer](https://arxiv.org/abs/1706.03762) in TensorFlow 2.\r\nCheckout my tutorial on [blog.tensorflow.org](https://blog.tensorflow.org/2019/05/transformer-chatbot-tutorial-with-tensorflow-2.html).\r\n\r\n### Updates\r\n- 16 June 2022:\r\n  - Update `setup.sh` script to install Apple Silicon version of TensorFlow 2.9 (only use this if you're feeling adventurous).\r\n  - Updated the two custom layers, `PositionalEncoding` and `MultiHeadAttentionLayer`, to allow model saving via `model.save()` or `tf.keras.models.save_model()`.\r\n  - `train.py` showcase how to call `model.save()` and `tf.keras.models.load_model()`.\r\n- 8 Dec 2020: Updated support to TensorFlow 2.3.1 and TensorFlow Datasets 4.1.0\r\n- 18 Jan 2020: Added [notebook](tf2_tpu_transformer_chatbot.ipynb) with  Google Colab TPU support in TensorFlow 2.1.\r\n\r\n### Packages\r\n- TensorFlow 2.9.1\r\n- [TensorFlow Datasets](https://www.tensorflow.org/datasets)\r\n\r\n### Setup\r\n- create new anaconda environment and initialize environment `chatbot`\r\n    ```\r\n    conda create -n chatbot python=3.8\r\n    conda activate chatbot\r\n    ```\r\n- run installation script\r\n    ```\r\n    sh setup.sh\r\n    ```\r\n- Note: the script would install CUDA and cuDNN via conda if installing on a Linux system, or `tensorflow-metal` for devices with Apple Silicon (Note that there are tons of bugs with TensorFlow on Apple Silicon GPU, e.g. [Adam optimizer does not work](https://developer.apple.com/forums/thread/691917)).\r\n\r\n### Dataset\r\n- We will use the conversations in movies and TV shows provided by [Cornell Movie-Dialogs Corpus](https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html), which contains more than 220 thousands conversational exchanges between more than 10k pairs of movie characters, as our dataset.\r\n- We pre-process our dataset in the following order:\r\n\t- Extract `max_samples` conversation pairs into list of `questions` and `answers`.\r\n\t- Pre-process each sentence by removing special characters in each sentence.\r\n\t- Build tokenizer (map text to ID and ID to text) using [TensorFlow Datasets SubwordTextEncoder](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder).\r\n\t- Tokenize each sentence and add `start_token` and `end_token` to indicate the start and end of each sentence.\r\n\t- Filter out sentence that has more than `max_length` tokens.\r\n\t- Pad tokenized sentences to `max_length`\r\n- Check [dataset.py](transformer/dataset.py) implementation.\r\n\r\n### Model\r\n![transformer model plot](transformer.png)\r\n- check [model.py](transformer/model.py) for the implementation of Multi-Headed Attention, Positional Encoding and Transformer.\r\n\r\n### Run\r\n- check all available flags and hyper-parameters `python main.py --help`\r\n```\r\npython train.py --output_dir runs/save_model --batch_size 256 --epochs 50 --max_samples 50000\r\n```\r\n- the final trained model will be saved to `runs/save_model`.\r\n\r\n### Samples\r\n```\r\ninput:\t\twhere have you been?\r\noutput:\t\ti m not talking about that .\r\n\r\ninput:\t\tit's a trap!\r\noutput:\t\tno , it s not .\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanlimy%2Ftf2-transformer-chatbot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbryanlimy%2Ftf2-transformer-chatbot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanlimy%2Ftf2-transformer-chatbot/lists"}