{"id":23513182,"url":"https://github.com/lyz508/gpt2-ai-writing","last_synced_at":"2026-05-15T22:03:45.533Z","repository":{"id":134231913,"uuid":"502697690","full_name":"lyz508/GPT2-AI-writing","owner":"lyz508","description":"Self-trained GPT-2 model, NYCU 2022 AI Final Project","archived":false,"fork":false,"pushed_at":"2022-06-17T15:24:05.000Z","size":3849,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-13T02:04:28.291Z","etag":null,"topics":["gpt-2","machine-learning","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lyz508.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-12T18:33:17.000Z","updated_at":"2024-12-25T12:06:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"17aa30b6-101e-4376-81d3-2d981df25a9a","html_url":"https://github.com/lyz508/GPT2-AI-writing","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/lyz508/GPT2-AI-writing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyz508%2FGPT2-AI-writing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyz508%2FGPT2-AI-writing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyz508%2FGPT2-AI-writing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyz508%2FGPT2-AI-writing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lyz508","download_url":"https://codeload.github.com/lyz508/GPT2-AI-writing/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lyz508%2FGPT2-AI-writing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279016357,"owners_count":26085828,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gpt-2","machine-learning","tensorflow"],"created_at":"2024-12-25T13:43:09.121Z","updated_at":"2025-10-13T17:35:30.053Z","avatar_url":"https://github.com/lyz508.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Writing with AI\r\n\r\n## Purpose\r\n\r\n- NYCU 2022 spring AI Final Project\r\n\r\n- The main purpose is to use the GPT-2 model trained \u0026 generating text\r\n\r\n## Requirement\r\n\r\n```\r\nattr: 0.3.1\r\nattrs: 21.4.0\r\nmatplotlib: 3.5.1\r\ntensorflow: 2.9.1\r\ntokenizers: 0.12.1\r\ntransformers: 4.19.4\r\n```\r\n\r\n## Simple Use\r\n\r\n- Download the release.\r\n\r\n- Follow the step below to construct directories.\r\n\r\n- Put files to the right position\r\n\r\n- Train model\r\n\r\n  ```shell\r\n  python train.py\r\n  ```\r\n\r\n- Write with the model\r\n\r\n\t```shell\r\n\tpython write.py [--dir \u003cmodel_path\u003e] [--max_len \u003cexpected_len\u003e]\r\n\t```\r\n\r\n## Code Structure\r\n\r\n### Structure of Program\r\n\r\n- Structure of the program\r\n\r\n\u003cimg src=\"media/sample-code-structure-before.jpg\" alt=\"image-20220613190118332\"  /\u003e\r\n\r\n- files\r\n\r\n  - `src/*`\r\n\r\n    - `config.py`: Store program configuration and will be instantiated in `train.py` or anywhere it need to be called.\r\n\r\n    - `model.py`: Including model initialize, train, save, visualize and log output. A combination of core functions.\r\n\r\n    - `tokenization.py`: Use this to trained an BPE tokenizer.\r\n\r\n      ```shell\r\n      python tokenization.py\r\n      ```\r\n\r\n      \u003e This need to be run if there are no corresponding tokenizer.\r\n\r\n  - `train.py`\r\n\r\n    - It will load tokenizer, build model, setup all project configuration and start training the module.\r\n\r\n  - `write.py`\r\n\r\n    - It can be used to generate text with existed models, which will stored in trained_model directory.\r\n\r\n### Mkdir\r\n\r\n- Some directories may need to be constructed before the program runs\r\n\r\n  ```shell\r\n  mkdir trained_data\r\n  mkdir tokenized_data\r\n  mkdir trained_model\r\n  ```\r\n\r\n  \u003e make sure to put data willing to train under the trained_data directory\r\n\r\n- An example structure with the provided pretrained model and put the data will be like.\r\n\r\n  \u003cimg src=\"media/sample-code-structure-release.png\" alt=\"sample\" style=\"zoom:80%;\" /\u003e\r\n\r\n  \r\n\r\n### Modify the config\r\n\r\n- Some codes may need to be modified for local use\r\n\r\n- `train.py`\r\n\r\n  ```python\r\n  \"\"\" Metadata\r\n  ...\r\n  \"\"\"\r\n  # ...\r\n  \r\n  config = ProjectConfig(\r\n  \t...,\r\n  \tdata_name=\"simplebooks-2\"\r\n  )\r\n  ```\r\n\r\n  \u003e data_name can be modified\r\n\r\n## Preprocessing\r\n\r\n### BPE Tokenizer\r\n\r\n- implement BPE tokenizer to pre-processing the text data\r\n\r\n  - [Summary of the tokenizers (huggingface.co)](https://huggingface.co/docs/transformers/tokenizer_summary)\r\n  - Aim's to translate between human-readable text and numeric indices\r\n  - Indices will be mapped to word embeddings (numerical representations of words) -\u003e This will be done by an embedding layer within the model. \r\n\r\n- Load the tokenizer\r\n\r\n  - This Tokenizer will be loaded in model initialization\r\n\r\n    ```python\r\n    tokenizer = GPT2Tokenizer.from_pretrained(tokenizer_path)\r\n    ```\r\n\r\n## Training\r\n\r\n### TFGPT2LMHeadModel\r\n\r\n- Use `transformers` to construct GPT Model\r\n\r\n### History\r\n\r\n- Stored history object will be used to visualize the trainning history.\r\n- Use `matplotlib.pyplot` to visualized data\r\n\r\n### Validation\r\n\r\n- Validify the train model with test model loss value \r\n- The train dataset is 10 times larger then the test dataset and there are no intersection of them\r\n- A way to test and modify the hyper-parameters\r\n\r\n![](media/simplebooks-2-train-50-test-epoch.png)\r\n\u003e It can be found that the loss value have no fitting problem\r\n\r\n![](media/simplebooks-2-50-train-test-batch.png)\r\n\u003e Since the training dataset is more larger than the test dataset, on the same number of batches, training model will have lower loss value\r\n\r\n\r\n## Result\r\n\r\n- This visualization result will be stored in `trained_model/figure/`\r\n- Detail log output can be found in `media/detail_output.md`\r\n\r\n### Text Generation\r\n\r\n![](media/Result.png)\r\n\r\n### Loss Value Per Epoch\r\n\r\n\u003cimg src=\"media/samplebooks2-50-epoch.png\" alt=\"loss per epoch\" style=\"zoom:80%;\" /\u003e\r\n\r\n### Loss Value Per Batch\r\n\r\n\u003cimg src=\"media/samplebooks2-50-batch.jpg\" alt=\"loss per batch\" style=\"zoom:80%;\" /\u003e\r\n\r\n\r\n## Performance\r\n- Comparison the performence with other project, which is also training the GPT2 model\r\n![baseline_comparison](media/baseline_comparison.png)\r\n- The way to retrive the baseline parameter is described in the \"baseline\" branch\r\n\r\n\r\n## Different Normalizer\r\n\r\n![simplebooks2-30-diff-normalizer](media/simplebooks2-30-diff-normalizer.png)\r\n\r\n\r\n\r\n## Reference\r\n\r\n- [Text generation with GPT-2 - Model Differently](https://www.modeldifferently.com/en/2021/12/generación-de-fake-news-con-gpt-2/)\r\n\r\n- [Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. (github.com)](https://github.com/huggingface/transformers)\r\n- [TensorFlow API](https://www.tensorflow.org/api_docs/python/tf?hl=zh-tw)\r\n- [Model training APIs (keras.io)](https://keras.io/api/models/model_training_apis/)\r\n- [TFGPT2LMHeadModel (huggingface.co)](https://huggingface.co/docs/transformers/v4.19.4/en/model_doc/gpt2#transformers.TFGPT2LMHeadModel)\r\n- [TFPreTrainedModel (huggingface.co)](https://huggingface.co/docs/transformers/v4.19.4/en/main_classes/model#transformers.TFPreTrainedModel)\r\n- [tf.keras.callbacks.History  | TensorFlow Core v2.9.1](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/History)\r\n- [tf.keras.callbacks.Callback  | TensorFlow Core v2.9.1](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback)\r\n- [Pipelines (huggingface.co)](https://huggingface.co/docs/transformers/v4.19.4/en/main_classes/pipelines#transformers.TextGenerationPipeline)\r\n- [Difference between Sparse Cross Entropy and Categorical Cross Entropy](https://ithelp.ithome.com.tw/articles/10271081)\r\n\r\n- [gpt2 · Hugging Face](https://huggingface.co/gpt2)\r\n- [Visualize the hyperparameter tuning process (keras.io)](https://keras.io/guides/keras_tuner/visualize_tuning/)\r\n- [python - How to disable printing reports after each epoch in Keras? - Stack Overflow](https://stackoverflow.com/questions/44931689/how-to-disable-printing-reports-after-each-epoch-in-keras)\r\n- [Module: tf.keras.metrics  | TensorFlow Core v2.9.1](https://www.tensorflow.org/api_docs/python/tf/keras/metrics)\r\n- [machine learning - What does from_logits=True do in SparseCategoricalcrossEntropy loss function? - Data Science Stack Exchange](https://datascience.stackexchange.com/questions/73093/what-does-from-logits-true-do-in-sparsecategoricalcrossentropy-loss-function)\r\n- [How to add some new special tokens to a pretrained tokenizer?](https://github.com/huggingface/tokenizers/issues/247)\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flyz508%2Fgpt2-ai-writing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flyz508%2Fgpt2-ai-writing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flyz508%2Fgpt2-ai-writing/lists"}