{"id":16544453,"url":"https://github.com/hsm207/bert_attn_viz","last_synced_at":"2026-03-06T20:31:49.582Z","repository":{"id":115473241,"uuid":"166560360","full_name":"hsm207/bert_attn_viz","owner":"hsm207","description":"Visualize BERT's self-attention layers on text classification tasks","archived":false,"fork":false,"pushed_at":"2019-03-07T16:49:35.000Z","size":235,"stargazers_count":47,"open_issues_count":1,"forks_count":12,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-09-06T06:43:23.885Z","etag":null,"topics":["attention","bert","explainable-ai","nlp","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hsm207.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-19T15:15:04.000Z","updated_at":"2024-11-27T15:11:29.000Z","dependencies_parsed_at":null,"dependency_job_id":"f920893f-f1a8-4951-91f3-c45c6059c81f","html_url":"https://github.com/hsm207/bert_attn_viz","commit_stats":{"total_commits":75,"total_committers":21,"mean_commits":"3.5714285714285716","dds":0.6933333333333334,"last_synced_commit":"3743d46ac2d1296096dff595385e371024d658fe"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hsm207/bert_attn_viz","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsm207%2Fbert_attn_viz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsm207%2Fbert_attn_viz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsm207%2Fbert_attn_viz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsm207%2Fbert_attn_viz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hsm207","download_url":"https://codeload.github.com/hsm207/bert_attn_viz/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hsm207%2Fbert_attn_viz/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30196173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T19:07:06.838Z","status":"ssl_error","status_checked_at":"2026-03-06T18:57:34.882Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","bert","explainable-ai","nlp","tensorflow"],"created_at":"2024-10-11T19:02:46.114Z","updated_at":"2026-03-06T20:31:49.551Z","avatar_url":"https://github.com/hsm207.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nThis repository is an adaptation of  the [bert](https://github.com/google-research/bert) repository.\n\nThe purpose of this repository is to visualize BERT's self-attention weights after it has been fine-tuned on the [IMDb dateset](http://ai.stanford.edu/~amaas/data/sentiment/). However, it can be extended to any text classification dataset by creating an appropriate DataProcessor class. See [run_classifier.py](run_classifier.py) for details.\n\n# Usage\n\n1. Create a tsv file for each of the IMDB training and test set.\n\n    Refer to the [imdb_data](https://github.com/hsm207/imdb_data) repo for instructions.\n\n2. Fine-tune BERT on the IMBDb training set.\n\n   Refer to the official BERT repo for fine-tuning instructions.\n    \n   Alternatively, you can skip this step by downloading the fine-tuned model from [here](https://drive.google.com/open?id=13Ajyk6xejy3kRU7Ewo_5slCo9db2bOdk).\n   \n   The pre-trained model (BERT base uncased) used to perform the fine-tuning can also be downloaded from [here](https://drive.google.com/open?id=1f23aE84MlPY1eQqzyENt4Fk_DGucof_4). \n\n3. Visualize BERT's weights.\n   \n   Refer to the [BERT_viz_attention_imdb](/bert_attn_viz/notebooks/BERT_viz_attention_imdb.ipynb) notebook for more details.\n \n# How it works\nThe forward pass has been modified to return a list of `{layer_i: layer_i_attention_weights}` dictionaries. The shape of \n `layer_i_attention_weights` is `(batch_size, num_multihead_attn, max_seq_length, max_seq_length)`.\n \n You can specify a function to process the above list by passing it as a parameter into the `load_bert_model` function\n in the [explain.model](explain/model.py) module. The function's output is avaialble as part of the result of the Estimator's predict call\n under the key named 'attention'.\n \n Currently, only two attention processor functions have been defined, namely `average_last_layer_by_head` and  `average_first_layer_by_head`.\n See [explain.attention](explain/attention.py) for implementation details.       \n# Model Performance Metrics\n\nThe fine-tuned model achieved an accuracy of 0.9407 on the test set. \n\nThe fine-tuning process was done with the following hyperparameters:\n\n* maximum sequence length: 512\n* training batch size: 8\n* learning rate: 3e-5\n* number of epochs: 3\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsm207%2Fbert_attn_viz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhsm207%2Fbert_attn_viz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhsm207%2Fbert_attn_viz/lists"}