{"id":21674168,"url":"https://github.com/abhimishra91/transformers-tutorials","last_synced_at":"2025-05-16T13:05:43.572Z","repository":{"id":41299790,"uuid":"261098554","full_name":"abhimishra91/transformers-tutorials","owner":"abhimishra91","description":"Github repo with tutorials to fine tune transformers for diff NLP tasks","archived":false,"fork":false,"pushed_at":"2024-04-01T14:43:18.000Z","size":289,"stargazers_count":846,"open_issues_count":19,"forks_count":191,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-12T08:38:35.514Z","etag":null,"topics":["bert","classification","deep-learning","distilbert","named-entity-recognition","natural-language-processing","nlp","pytorch","pytorch-tutorial","t5","transformers","wandb"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abhimishra91.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-04T06:37:14.000Z","updated_at":"2025-04-10T14:45:43.000Z","dependencies_parsed_at":"2024-11-25T16:02:45.733Z","dependency_job_id":null,"html_url":"https://github.com/abhimishra91/transformers-tutorials","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhimishra91%2Ftransformers-tutorials","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhimishra91%2Ftransformers-tutorials/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhimishra91%2Ftransformers-tutorials/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abhimishra91%2Ftransformers-tutorials/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abhimishra91","download_url":"https://codeload.github.com/abhimishra91/transformers-tutorials/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254535827,"owners_count":22087399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","classification","deep-learning","distilbert","named-entity-recognition","natural-language-processing","nlp","pytorch","pytorch-tutorial","t5","transformers","wandb"],"created_at":"2024-11-25T13:43:19.021Z","updated_at":"2025-05-16T13:05:43.523Z","avatar_url":"https://github.com/abhimishra91.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003ch1 align=\"center\"\u003ePyTorch Transformers Tutorials \u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg alt=\"Transformer Tutorials\" src=\"meta/banner.png\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://github.com/abhimishra91/transformers-tutorials/issues\"\u003e\u003cimg alt=\"GitHub issues\" src=\"https://img.shields.io/github/issues/abhimishra91/transformers-tutorials\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/abhimishra91/transformers-tutorials/network\"\u003e\u003cimg alt=\"GitHub forks\" src=\"https://img.shields.io/github/forks/abhimishra91/transformers-tutorials\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/abhimishra91/transformers-tutorials/stargazers\"\u003e\u003cimg alt=\"Github Stars\" src=\"https://img.shields.io/github/stars/abhimishra91/transformers-tutorials\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/abhimishra91/transformers-tutorials/blob/master/LICENSE\"\u003e\u003cimg alt=\"GitHub license\" src=\"https://img.shields.io/github/license/abhimishra91/transformers-tutorials\"\u003e\u003c/a\u003e\n\n\n### Introduction\n\nThe field of **NLP** was revolutionized in the year 2018 by introduction of **BERT** and his **Transformer** friends(RoBerta, XLM etc.).\n\nThese novel transformer based neural network architectures and new ways to training a neural network on natural language data introduced transfer learning to NLP problems. Transfer learning had been giving out state of the art results in the Computer Vision domain for a few years now and introduction of transformer models for NLP brought about the same paradigm change in NLP.\n\nCompanies like [Google](https://github.com/google-research/bert) and [Facebook](https://github.com/pytorch/fairseq/tree/master/examples/roberta) trained their neural networks on large swathes of Natural Language Data to grasp the intricacies of language thereby generating a  Language model. Finally these models were fine tuned to specific domain dataset to achieve state of the art results for a specific problem statement. They also published these trained models to open source community. The community members were now able to fine tune these models to their specific use cases.\n\n[Hugging Face](https://github.com/huggingface) made it easier for community to access and fine tune these models using their Python Package: [Transformers](https://github.com/huggingface/transformers). \n\n### Motivation\nDespite these amazing technological advancements applying these solutions to business problems is still a challenge given the niche knowledge required to understand and apply these method on specific problem statements. Hence, In the following tutorials i will be demonstrating how a user can leverage technologies along with some other python tools to fine tune these Language models to specific type of tasks. \n\nBefore i proceed i will like to mention the following groups for the fantastic work they are doing and sharing which have made these notebooks and tutorials possible:\n\nPlease review these amazing sources of information and subscribe to their channels/sources.\n - [Hugging Face Team](https://huggingface.co/)\n - Abhishek Thakur for his amazing [Youtube videos](https://www.youtube.com/user/abhisheksvnit)\n\nThe problem statements that i will be working with are:\n\n| Notebook |Github Link |Colab Link|Kaggle Kernel|\n|--|--|--|--|\n|Text Classification: Multi-Class| [Github](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb) |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb)|[Kaggle](https://www.kaggle.com/eggwhites2705/transformers-multiclass-classification-ipynb)|\n|Text Classification: Multi-Label| [Github](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)|[Kaggle](https://www.kaggle.com/eggwhites2705/transformers-multi-label-classification)|\n|Sentiment Classification **with Experiment Tracking in [WandB](https://app.wandb.ai/abhimishra-91/transformers_tutorials_sentiment/runs/1zwn4gbg?workspace=user-abhimishra-91)!**|[Github](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_sentiment_wandb.ipynb)|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_sentiment_wandb.ipynb)||\n|Named Entity Recognition: **with TPU processing!**|[Github](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_ner.ipynb)|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_ner.ipynb)|[Kaggle](https://www.kaggle.com/eggwhites2705/transformers-ner)|\n|Question Answering||||\n|Summary Writing: **with Experiment Tracking in [WandB](https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization?workspace=user-abhimishra-91)!**|[Github](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)|[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)|[Kaggle](https://www.kaggle.com/eggwhites2705/transformers-summarization-t5/output)|\n\n\n\n### Directory Structure\n\n 1. `data`: This folder contains all the toy data used for fine tuning. \n 2. `utils`: This folder will contain any miscellaneous script used to prepare for the fine tuning.\n 3. `models`: Folder to save all the artifacts post fine tuning. \n\n### Further Watching/Reading\n\nI will try to cover the practical and implementation aspects of fine tuning of these language models on various NLP tasks. You can improve your knowledge on this topic by reading/watching the following resources.\n\n\n - Watching\n\t - [Introduction in Simple terms](https://www.youtube.com/watch?v=gcHkxP9adiM)\n\t - [Transfer Learning in NLP](https://www.youtube.com/watch?v=0T_Qr4qBrqc)\n\t - [BERT Research Series from ChrisMcCormickAI](https://www.youtube.com/playlist?list=PLam9sigHPGwOBuH4_4fr-XvDbe5uneaf6)\n\n- Reading\n\t - [Transformers Documentation](https://huggingface.co/transformers/)\n\t - [Pytorch Documentation](https://pytorch.org/docs/stable/index.html)\n\t - [Google AI Blog](https://ai.googleblog.com/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhimishra91%2Ftransformers-tutorials","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabhimishra91%2Ftransformers-tutorials","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabhimishra91%2Ftransformers-tutorials/lists"}