{"id":17881913,"url":"https://github.com/ztjhz/minilm","last_synced_at":"2025-07-30T11:07:03.962Z","repository":{"id":206626061,"uuid":"710664330","full_name":"ztjhz/miniLM","owner":"ztjhz","description":"Small Model Is All You Need - NTU SC4001 Neural Network \u0026 Deep Learning Project","archived":false,"fork":false,"pushed_at":"2023-11-09T18:55:25.000Z","size":16468,"stargazers_count":17,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-27T01:50:49.431Z","etag":null,"topics":["bert","deep-learning","deepspeed","gpt2","llama","llm","neural-network","nlp","ntu","roberta","sc4001","wandb"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ztjhz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-10-27T07:13:21.000Z","updated_at":"2025-02-17T13:05:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"72d2e216-059c-4e49-a935-9e79c2995516","html_url":"https://github.com/ztjhz/miniLM","commit_stats":{"total_commits":58,"total_committers":1,"mean_commits":58.0,"dds":0.0,"last_synced_commit":"b74d927ae5486c627296059c7cf2ab0ce0fef166"},"previous_names":["ztjhz/minilm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ztjhz/miniLM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztjhz%2FminiLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztjhz%2FminiLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztjhz%2FminiLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztjhz%2FminiLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ztjhz","download_url":"https://codeload.github.com/ztjhz/miniLM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ztjhz%2FminiLM/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267442904,"owners_count":24087893,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-27T02:00:11.917Z","response_time":82,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","deep-learning","deepspeed","gpt2","llama","llm","neural-network","nlp","ntu","roberta","sc4001","wandb"],"created_at":"2024-10-28T12:46:01.011Z","updated_at":"2025-07-30T11:07:03.904Z","avatar_url":"https://github.com/ztjhz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Small Model Is All You Need\n\nWelcome to the official repository for the NTU SC4001 Neural Network \u0026 Deep Learning project report _\"Small Model Is All You Need\"_. This repository contains all the code and resources required to replicate the findings and experiments presented in the report.\n\n\u003cp align='center'\u003e\n  \u003cimg src=\"public/pic.png\" width=250 /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/ztjhz/miniLM/blob/main/public/Report.pdf\"\u003eProject Report\u003c/a\u003e\n    |\n    \u003ca href=\"https://wandb.ai/sc4001/text-sentiment-analysis/\"\u003eW\u0026B Runs\u003c/a\u003e\n    |\n    \u003ca href=\"https://api.wandb.ai/links/sc4001/vz711n0m\"\u003eW\u0026B Graphs\u003c/a\u003e\n\u003c/p\u003e\n\nThe rise of large language models (LLMs) such as GPT-3 and its successors has heralded a new era in natural language processing (NLP), offering unprecedented capabilities across a wide spectrum of tasks. However, the universal application of these computational giants for all NLP tasks has been questioned, particularly when more straightforward tasks may not necessitate the heft and complexity of LLMs.\n\nThis report investigates the performance of large and small language models on sentiment analysis tasks, aiming to ascertain whether such tasks genuinely benefit from the scale of LLMs:\n\n\u003cp align='center'\u003e\n  \u003cimg src=\"public/sliced-model.png\" width=500 /\u003e\n\u003c/p\u003e\n\n- Through a novel slicing technique applied to the Llama model, we reveal that not all layers contribute equally, with middle layers often outperforming others.\n- We extend our analysis to RoBERTa, contrasting the effectiveness of pretraining against fine-tuning on smaller datasets such as IMDb.\n- Additionally, we fine-tune smaller models like RoBERTa, GPT-2, and T5, which demonstrate results comparable with LLMs across several benchmarks, including IMDb, SST-2, and Yelp datasets.\n- We also did a further analysis on slicing RoBERTa and discovered that RoBERTa's capabilities is best realized through fine-tuning across all layers due to its less capable zero-shot abilities.\n\nOur findings advocate for a more nuanced selection of language models, demonstrating that smaller models can be equally adept as LLMs for certain applications, thus providing a more cost-effective and resource-efficient alternative. This project aims to guide companies and startups in choosing the right model for the right task, highlighting that while LLMs hold substantial value, they are not a one-size-fits-all solution in the realm of AI-driven language tasks.\n\n- [Environment Setup](#environment-setup)\n  - [Setting Up a Python Virtual Environment (Optional)](#setting-up-a-python-virtual-environment-optional)\n  - [Installing Dependencies](#installing-dependencies)\n- [Training](#training)\n  - [Training Scripts](#training-scripts)\n  - [Selecting a GPU Device](#selecting-a-gpu-device)\n  - [Script Options](#script-options)\n  - [Handling CUDA Out of Memory Errors](#handling-cuda-out-of-memory-errors)\n- [Authors](#authors)\n- [Project Report](#project-report)\n\n## Environment Setup\n\nBefore you begin, make sure your development environment is set up correctly.\n\n### Setting Up a Python Virtual Environment (Optional)\n\nIt's a good practice to create a virtual environment to manage dependencies for your project. To set up a Python virtual environment, run:\n\n```sh\npython3 -m venv .venv\nsource .venv/bin/activate\n```\n\n### Installing Dependencies\n\nInstall all required packages using `pip`:\n\n```sh\npip install -r requirements.txt\n```\n\nYou will also need to install PyTorch. Follow the instructions specific to your platform at [PyTorch Get Started](https://pytorch.org/get-started/locally/).\n\n## Training\n\nOur project leverages [DeepSpeed](https://github.com/microsoft/DeepSpeed) to optimize training on GPU. As such, we currently only support training on GPU-enabled setups.\n\n### Training Scripts\n\nTraining scripts are provided in the `/scripts` directory. To start training:\n\n1. Navigate to the base directory of the project.\n2. Run the appropriate script for your training needs.\n\n**Example Commands:**\n\n- To run the Sliced Llama training:\n\n  ```sh\n  sh scripts/llama.sh\n  ```\n\n- To run the Comparing Transformers training:\n  ```sh\n  sh scripts/compare_transformers.sh\n  ```\n\n### Selecting a GPU Device\n\nYou can specify the GPU device(s) to use with the `--include` flag followed by the device IDs.\n\nExample: `--include localhost:2,3` will use GPU devices 2 and 3.\n\n### Script Options\n\nBelow are the options available in various scripts:\n\n**Comparing Transformers Script Options:**\n\n```sh\n--dataset        Choose from 'imdb', 'yelp', 'sst2'. Default is 'imdb'.\n--model          Choose from 'roberta', 'gpt2', 't5'. Default is 'roberta'.\n--subset_yelp    Subset the Yelp dataset to reduce training time. Default is False.\n```\n\n**Small Dataset Script Options:**\n\n```sh\n--init       'train' to start training from scratch, 'finetune' to start from pre-trained weights.\n--model      The model to use. Default is 'roberta'.\n--run_name   The run name for the Weights \u0026 Biases experiment.\n```\n\n**Llama Script Options:**\n\n```sh\n--dataset    Choose from 'imdb', 'yelp'. Default is 'imdb'.\n```\n\n### Handling CUDA Out of Memory Errors\n\nIf you encounter a CUDA out of memory error during training, try adjusting the `allgather_bucket_size` and `reduce_bucket_size` parameters in the `ds_config.json` file.\n\n## Authors\n\n\u003ca href='https://github.com/ztjhz' title='Toh Jing Hua'\u003e \u003cimg src='https://avatars.githubusercontent.com/ztjhz' height='75' width='75'/\u003e\u003c/a\u003e\n\u003ca href='https://github.com/xJQx' title='Toh Jing Qiang'\u003e \u003cimg src='https://avatars.githubusercontent.com/xJQx' height='75' width='75'/\u003e\u003c/a\u003e\n\u003ca href='https://github.com/Lebarnon' title='Lenson Lim'\u003e \u003cimg src='https://avatars.githubusercontent.com/Lebarnon' height='75' width='75'/\u003e\u003c/a\u003e\n\n## Project Report\n\nFor a comprehensive understanding of our project, methodologies, and detailed results, please refer to our project report. You can access the report at the following link: [Project Report - Small Model Is All You Need](https://github.com/ztjhz/miniLM/blob/main/public/Report.pdf)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fztjhz%2Fminilm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fztjhz%2Fminilm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fztjhz%2Fminilm/lists"}