{"id":21075305,"url":"https://github.com/burcgokden/sdpa-transformer-wrapper","last_synced_at":"2026-04-21T20:06:55.010Z","repository":{"id":260621933,"uuid":"380477092","full_name":"burcgokden/SDPA-Transformer-Wrapper","owner":"burcgokden","description":"A framework for training and evaluating a transformer with scaled dot product attention on a tensorflow dataset.","archived":false,"fork":false,"pushed_at":"2021-07-19T06:23:27.000Z","size":28,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-31T10:05:32.711Z","etag":null,"topics":["attention","deep-learning","embeddings","machine-translation","natural-langugage-processing","neural-network","nlp","nmt","scaled-dot-product","smt","tensorflow","tensorflow-datasets","tensorflow-text","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/burcgokden.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-26T10:41:09.000Z","updated_at":"2021-07-19T06:23:30.000Z","dependencies_parsed_at":"2024-11-01T12:32:21.252Z","dependency_job_id":"cc55dafa-7286-40b7-a7d9-b43d956db07a","html_url":"https://github.com/burcgokden/SDPA-Transformer-Wrapper","commit_stats":null,"previous_names":["burcgokden/sdpa-transformer-wrapper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/burcgokden/SDPA-Transformer-Wrapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burcgokden%2FSDPA-Transformer-Wrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burcgokden%2FSDPA-Transformer-Wrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burcgokden%2FSDPA-Transformer-Wrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burcgokden%2FSDPA-Transformer-Wrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/burcgokden","download_url":"https://codeload.github.com/burcgokden/SDPA-Transformer-Wrapper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burcgokden%2FSDPA-Transformer-Wrapper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32108196,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-21T11:25:29.218Z","status":"ssl_error","status_checked_at":"2026-04-21T11:25:28.499Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","deep-learning","embeddings","machine-translation","natural-langugage-processing","neural-network","nlp","nmt","scaled-dot-product","smt","tensorflow","tensorflow-datasets","tensorflow-text","transformer"],"created_at":"2024-11-19T19:21:21.857Z","updated_at":"2026-04-21T20:06:54.982Z","avatar_url":"https://github.com/burcgokden.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Transformer with Scaled Dot Product Attention\n\nThis repository is the implementation of Scaled Dot  Product Attention (SDPA) Transformer model within a wrapper code suite for training and evaluation. The details of the transformer model can be found in [Transformer model for language understanding](https://www.tensorflow.org/text/tutorials/transformer)\n\nThis training and evaluation suite was used to train and evaluate a SDPA transformer model for Turkish-English and Portuguese-English translation tasks in the research article: [Power Law Graph Transformer for Machine Translation and Representation Learning](https://arxiv.org/abs/2107.02039)\n\n#### Key Features\n\n- Flexible model customization through a hyperparameter dictionary for SDPA Transformer model parameters.\n- Simple interface for training the model with checkpoints at custom intervals, and highest accuracy observed.\n- Early stopping after a number of epochs based on validation loss.\n- Simple interface for evaluating trained model using BLEU score with greedy search and beam search.\n- Data preparation framework for Neural Machine Translation for tensorflow datasets with capability to use a percentage of the train dataset or filter dataset based on a token  number in a sentence. \n- Capability to reverse source and target languages for input dataset.\n- Keeps track of train and validation loss/accuracy for each epoch.\n\n#### Sample Run:\n\nSample run trains and evaluates a 4-layer 8-head SDPA Transformer model for PT-EN translation task from tensorflow dataset found at: [ted_hrlr_translate/pt_to_en](https://www.tensorflow.org/datasets/catalog/ted_hrlr_translate#ted_hrlr_translatept_to_en)\n\nThe tokenizer model is built using BERT Subword Tokenizer for Machine Translation implemented at [BERT Subword Tokenizer for Machine Translation](https://github.com/burcgokden/BERT-Subword-Tokenizer-Wrapper)\n\n- Prepare the Dataset for Machine Translation:\n\n```python\nimport nmt_data_prep as ndp\n\ninp_obj = ndp.src_tgt_data_prep(\n                 src_lang='pt',\n                 tgt_lang='en',\n                 BUFFER_SIZE=20000,\n                 BATCH_SIZE = 64,\n                 dataset_file=\"ted_hrlr_translate/pt_to_en\",\n                 load_dataset=True,\n                 train_percent=None,\n                 model_name = \"./ted_hrlr_translate_pt_en_tokenizer\",\n                 revert_order=False,\n                 shuffle_set=True,\n                 shuffle_files=True,\n                 MAX_LENGTH=None, \n                 verbose=True)\n```\n\n- Define hyperparameter dictionary for SDPA Transformer:\n\n```python\nhpdict_sdpa_transformer = {\n          \"num_layers\":4,\n          \"d_model\": 512,\n          \"num_heads\": 8,\n          \"dropout_rate\": 0.1,\n          \"dff\": 2048,\n          \"input_vocab_size\": inp_obj.tokenizers_src.get_vocab_size(),\n          \"target_vocab_size\": inp_obj.tokenizers_tgt.get_vocab_size(),\n          \"pe_input\": 1000,\n          \"pe_target\": 1000,\n          \"epochs\": 40,\n          \"save_model_path\": \"my_sdpa_transformer\",       \n          \"early_stop_threshold\": 4.0,\n          \"early_stop_counter\": 10,\n          \"early_stop_accuracy\": 0.59,\n          \"warmup_steps\": 4000\n          }\n```\n\n- Initialize the end-to-end model training suite and run train:\n```python\nimport sdpa_transformer_run_model as sdpa_run\n\ne2e_obj=sdpa_run.sdpa_transformer_e2e(\n                                      tokenizer_obj_src = inp_obj.tokenizers_src,\n                                      tokenizer_obj_tgt = inp_obj.tokenizers_tgt,\n                                      checkpoint_path = './model_saves/',\n                                      hpdict=hpdict_sdpa_transformer ,\n                                      load_ckpt=None,\n                                     )\n\ntrain_loss, train_accuracy, val_loss, val_accuracy=e2e_obj.train_model(\n                                                                       inp_obj.train_batches, \n                                                                       inp_obj.val_batches,\n                                                                       chkpt_epochs=[24, 30]\n                                                                      )\n\n```\n\n- Evaluate the trained SDPA Model using greedy or beam search:\n\n```python\nimport sdpa_evaluate_bleu_score as sebg\n\n#greedy search only\nsebg.evaluate_bleu(\n                 model_dict=hpdict_sdpa_transformer,\n                 model_name=\"my_sdpa_transformer\",\n                 model_type='train',\n                 src_lang='pt',\n                 tgt_lang='en',\n                 dataset_file=\"ted_hrlr_translate/pt_to_en\",\n                 revert_order=False,\n                 inp_obj=None,\n                 chkpt_path= './model_saves/',\n                 data_path= './model_data/',              \n                 load_ckpt='train', # 'val' | 'valacc' | custom checkpoint path\n                 tok_model_name=\"./ted_hrlr_translate_pt_en_tokenizer\",\n                 max_length=50,  #offset to evaluate model beyond input sentence length\n                 ds_max_length=None, #None for no filtering input sentence length\n                 verbose=True\n                )\n\n#beam search\nsebg.beam_evaluate_bleu(\n                 model_dict=hpdict_sdpa_transformer,\n                 beam_size=4,\n                 model_name=\"my_sdpa_transformer\",\n                 model_type='train',\n                 src_lang='pt',\n                 tgt_lang='en',\n                 dataset_file=\"ted_hrlr_translate/pt_to_en\",\n                 revert_order=False,\n                 inp_obj=None,\n                 chkpt_path= './model_saves/',\n                 data_path= './model_data/',              \n                 load_ckpt='train',\n                 tok_model_name=\"./ted_hrlr_translate_pt_en_tokenizer\",\n                 max_length=50,\n                 ds_max_length=None,\n                 verbose=True\n                )\n\n```\n\n#### Loss and Accuracy Curves:\n\nTrain and validation loss/accuracy values for each epoch are saved as pickle file and can be found in the train folder under save_model_path name:\n\n```python\nimport common as cm\n\ntrain_loss=cm.pklload(\"./model_saves/train/my_sdpa_transformer/train_loss.pkl\")\nval_loss=cm.pklload(\"./model_saves/train/my_sdpa_transformer/val_loss.pkl\")\ntrain_acc=cm.pklload(\"./model_saves/train/my_sdpa_transformer/train_accuracies.pkl\")\nval_acc=cm.pklload(\"./model_saves/train/my_sdpa_transformer/val_accuracies.pkl\")\n```\n\n#### Single Instance Evaluation:\n\nA sentence can be translated and compared to ground truth using greedy search only or beam search methods for single instance evaluation:\n\n```python\ne2e_model=sdpa_run.sdpa_transformer_e2e(\n                                      tokenizer_obj_src = inp_obj.tokenizers_src,\n                                      tokenizer_obj_tgt = inp_obj.tokenizers_tgt,\n                                      checkpoint_path = './model_saves/',\n                                      hpdict=hpdict_sdpa_transformer ,\n                                      load_ckpt='train' # 'val' | 'valacc' | custom checkpoint path\n                                     )\n\n#greedy search only\ntranslated_text, translated_tokens, _, eval_length = e2e_model.evaluate(sentence, max_length=50)\ne2e_model.print_translation(sentence, translated_text, ground_truth, eval_length)\n\n#beam search\ntranslated_text_list, translated_tokens_list, tranlated_tokenid_list, eval_length = e2e_model.beam_evaluate(sentence, beam_size=4, max_length=50)\ne2e_model.print_translation(sentence, translated_text_list[0], ground_truth, eval_length)\n```\n\n- Below sentences from test dataset are evaluated with beam length=4 by a model trained with same hyperparameters for a SDPA transformer detailed in the research article. Evaluation output may vary with each newly trained model.\n\n\u003e **Translating from:** perdemos o medo de criar uma coisa nova .  \n\u003e **Best probable translation:** we lost fear to create something new .  \n\u003e **Ground Truth:** we lost the fear of creating something new .  \n\u003e\n\u003e **Translating from:** vou mostrar aqui alguns exemplos , e vamos examinar alguns deles .  \n\u003e **Best probable translation:** let me show you here some examples , and let ' s examine some of them .  \n\u003e **Ground Truth:** i 'm going to show you some examples here , and we will run through some of them .  \n\u003e\n\u003e **Translating from:** ok , hoje quero falar sobre a forma como falamos do amor .  \n\u003e **Best probable translation:** okay , today i want to talk about how we talk about love .  \n\u003e **Ground Truth:** ok , so today i want to talk about how we talk about love .  \n\u003e\n\u003e **Translating from:** mas há uma grande diferença , isso só acontece dentro da colónia .  \n\u003e **Best probable translation:** but there ' s a big difference , it just happens inside the colony .  \n\u003e **Ground Truth:** but there 's a big difference , which is that it only happens within the colony .  \n\u003e\n\u003e **Translating from:** mas muito bons a absorver informação de muitas fontes diversas ao mesmo tempo .  \n\u003e **Best probable translation:** but very good at absorbing data from a lot of different sources at the same time .  \n\u003e **Ground Truth:** but they 're very good at taking in lots of information from lots of different sources at once .  \n\u003e\n\u003e **Translating from:** não podia construir isto com um anel de aço , da forma que sabia .  \n\u003e **Best probable translation:** i could n ' t build this up with a steel ring in the way i knew .  \n\u003e **Ground Truth:** i could n't build this with a steel ring , the way i knew .  \n\u003e\n\u003e **Translating from:** e gostaria de continuar a construir monumentos , que são amados por pessoas .  \n\u003e **Best probable translation:** and i ' d like to continue building monuments , which are loved by people .  \n\u003e **Ground Truth:** and i 'd like to keep building monuments that are beloved by people .  \n\u003e\n\u003e **Translating from:** a questão é que temos que ter um contexto , um limite para as nossas ações em tudo isto .  \n\u003e **Best probable translation:** the key thing is that we have to have a context , a range for our actions in all of this .  \n\u003e **Ground Truth:** the point is that we have to have a context , a gauge for our actions in all this .  \n\u003e\n\u003e **Translating from:** somos mais inteligentes , mais flexivéis , capazes de aprender mais , sobrevivemos em diferentes ambientes , emigrámos para povoar o mundo e viajámos até ao espaço .  \n\u003e **Best probable translation:** we ' re more intelligent , more flexible , we can learn more , we ' ve lived in different environments , we ' ve got into the world , and we push to space .  \n\u003e **Ground Truth:** we 're smarter , we 're more flexible , we can learn more , we survive in more different environments , we migrated to cover the world and even go to outer space .  \n\u003e\n\u003e **Translating from:** olhando para trás para os destroços e desespero daqueles anos , parece-me agora como se alguém tivesse morrido naquele lugar , e , no entanto , uma outra pessoa foi salva .  \n\u003e **Best probable translation:** looking behind the rubble and desperation of those years , it seems to me now as if someone died in that place , and yet another person was saved .  \n\u003e **Ground Truth:** now looking back on the wreckage and despair of those years , it seems to me now as if someone died in that place , and yet , someone else was saved .  \n\u003e\n\u003e **Translating from:** o cérebro pega em informação sem sentido e faz sentido a partir disso , o que significa que nunca vemos o que lá está , nunca vemos informação , só vemos o que nos foi útil ver no passado .  \n\u003e **Best probable translation:** so the brain takes information without it , and it makes sense out of it , which means that we never see what is there , we never see information , we only see what was useful for us to see in the past .  \n\u003e **Ground Truth:** right ? the brain takes meaningless information and makes meaning out of it , which means we never see what 's there , we never see information , we only ever see what was useful to see in the past .  \n\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fburcgokden%2Fsdpa-transformer-wrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fburcgokden%2Fsdpa-transformer-wrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fburcgokden%2Fsdpa-transformer-wrapper/lists"}