{"id":16039527,"url":"https://github.com/c00k1ez/plain-transformers","last_synced_at":"2025-10-09T17:09:07.709Z","repository":{"id":49563480,"uuid":"349500454","full_name":"c00k1ez/plain-transformers","owner":"c00k1ez","description":"Transformer models implementation for training from scratch.","archived":false,"fork":false,"pushed_at":"2022-12-28T13:05:30.000Z","size":59,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-21T10:36:34.034Z","etag":null,"topics":["artificial-intelligence","data-science","deep-learning","library","natural-language-processing","nlp","python","pytorch","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c00k1ez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-19T17:15:59.000Z","updated_at":"2024-02-11T14:09:18.000Z","dependencies_parsed_at":"2023-01-31T06:31:08.446Z","dependency_job_id":null,"html_url":"https://github.com/c00k1ez/plain-transformers","commit_stats":null,"previous_names":["c00k1ez/plain_transformers"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c00k1ez%2Fplain-transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c00k1ez%2Fplain-transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c00k1ez%2Fplain-transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c00k1ez%2Fplain-transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c00k1ez","download_url":"https://codeload.github.com/c00k1ez/plain-transformers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246150349,"owners_count":20731418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","data-science","deep-learning","library","natural-language-processing","nlp","python","pytorch","transformers"],"created_at":"2024-10-08T23:05:13.389Z","updated_at":"2025-10-09T17:09:07.613Z","avatar_url":"https://github.com/c00k1ez.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Simple way to use transformer models\n\n[![PyPI version](https://badge.fury.io/py/plain-transformers.svg)](https://badge.fury.io/py/plain-transformers)\n![Build Status](https://github.com/c00k1ez/plain_transformers/actions/workflows/ci_tests.yaml/badge.svg)\n\n## Features\n* Multimodal Transformer (text, text -\u003e text; for example for Automatic Post-Editing task);\n* Label smoothing loss;\n* Unlikelihood loss (sentence-level and for full context), [arxiv](https://arxiv.org/abs/1908.04319);\n* LayerDrop technique for transformer encoder/decoder layers, [arxiv](https://arxiv.org/abs/1909.11556);\n* Pre/Post LayerNorm encoders (Decoders in process), [arxiv](https://arxiv.org/abs/2004.08249);\n* ADMIN Initialization (in future), [arxiv](https://arxiv.org/abs/2004.08249);\n* Top-k/Top-p sampling with temperature, [arxiv](https://arxiv.org/abs/1904.09751);\n\n## Installation\n```pip install plain-transformers```\n\n## Usage\nMultimodal transformer example with two tokenizers:\n\n**Step one**: import model and some usefull staff;\n```python\nimport torch\n\nfrom plain_transformers.models import MultimodalTransformer\nfrom plain_transformers.layers import MultimodalTransformerDecoder\nfrom plain_transformers.layers import TransformerEncoder\n\nfrom plain_transformers import BPEWrapper\nfrom plain_transformers.initializations import normal_initialization, initialize_weights\nfrom plain_transformers.samplers.nucleus_sampler import NucleusSampler\n\nimport youtokentome as yttm\n```\n**Step two**: train and load tokenizers;\n```python\n# train your encoder tokenizer\nyttm.BPE.train(..., model='encoder_tokenizer.model')\n# train your decoder tokenizer\nyttm.BPE.train(..., model='decoder_tokenizer.model')\n\n# load tokenizers\nencoder_tokenizer = BPEWrapper(model='encoder_tokenizer.model')\ndecoder_tokenizer = BPEWrapper(model='decoder_tokenizer.model')\n```\n**Step three**: init out model configuration;\n```python\ncfg = {\n    'd_model': 768,\n    'first_encoder': {\n        'first_encoder_vocab_size': encoder_tokenizer.vocab_size(),\n        'first_encoder_max_length': 512,\n        'first_encoder_pad_token_id': encoder_tokenizer.pad_id,\n        'first_encoder_token_type_vocab_size': 2,\n        'first_encoder_n_heads': 8,\n        'first_encoder_dim_feedforward': 2048,\n        'first_encoder_num_layers': 3,\n        'first_encoder_type': 'post_ln'\n    },\n    'second_encoder': {\n        'second_encoder_vocab_size': encoder_tokenizer.vocab_size(),\n        'second_encoder_max_length': 512,\n        'second_encoder_pad_token_id': encoder_tokenizer.pad_id,\n        'second_encoder_token_type_vocab_size': 2,\n        'second_encoder_n_heads': 8,\n        'second_encoder_dim_feedforward': 2048,\n        'second_encoder_num_layers': 3,\n        'second_encoder_type': 'post_ln'\n    },\n    'decoder': {\n        'decoder_max_length': 512,\n        'decoder_vocab_size': decoder_tokenizer.vocab_size(),\n        'decoder_pad_token_id': decoder_tokenizer.pad_id,\n        'decoder_token_type_vocab_size': 2,\n        'decoder_n_heads': 8,\n        'decoder_dim_feedforward': 2048,\n        'decoder_num_layers': 3,\n        'decoder_type': 'post_ln'\n    },\n}\n```\n**Step four**: initialize model and apply weight initialisation (with default parameter ```std=0.02```);\n```python\nmodel = MultimodalTransformer(\n    TransformerEncoder,\n    TransformerEncoder,\n    MultimodalTransformerDecoder,\n    cfg['d_model'],\n    **cfg['first_encoder'],\n    **cfg['second_encoder'],\n    **cfg['decoder'],\n    share_decoder_head_weights=True,\n    share_encoder_decoder_embeddings=False,\n    share_encoder_embeddings=True,\n)\n\ninitialize_weights(model, normal_initialization, init_range=0.02)\n```\n**Step five**: train our model like ordinary seq2seq;\n```python\ntrain(model, ...)\n```\n**Step six**: initialize Sampler and generate model answer;\n```python\nsampler = NucleusSampler(model, encoder_tokenizer=(encoder_tokenizer, encoder_tokenizer), decoder_tokenizer=decoder_tokenizer)\nsampler.generate('Hello Bob, what are you doing?', second_input_text='Fine, thanks!', top_k=5)\n```\n## Example\nYou can find working example of NMT [here](https://colab.research.google.com/drive/1WA_CcmDD-O51foBvSOMKT4sZD-zuzQwA?usp=sharing).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc00k1ez%2Fplain-transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc00k1ez%2Fplain-transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc00k1ez%2Fplain-transformers/lists"}