{"id":21564984,"url":"https://github.com/mahtabranjbar/bert-model-benchmark-for-imdb-sentiment-classification","last_synced_at":"2025-10-08T09:56:19.334Z","repository":{"id":256858366,"uuid":"856641794","full_name":"MahtabRanjbar/BERT-Model-Benchmark-for-IMDB-Sentiment-Classification","owner":"MahtabRanjbar","description":"This project benchmarks various BERT-based models on the IMDB movie review dataset for sentiment classification, evaluating accuracy, precision, recall, and F1 score.","archived":false,"fork":false,"pushed_at":"2024-09-13T00:06:59.000Z","size":41,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-08T09:56:18.302Z","etag":null,"topics":["bert-fine-tuning","bert-models","classification","llms","nlp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MahtabRanjbar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-12T23:59:22.000Z","updated_at":"2025-08-05T18:44:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"a51c59ed-8f46-4027-8a63-f0761e70465d","html_url":"https://github.com/MahtabRanjbar/BERT-Model-Benchmark-for-IMDB-Sentiment-Classification","commit_stats":null,"previous_names":["mahtabranjbar/bert-model-benchmark-for-imdb-sentiment-classification"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MahtabRanjbar/BERT-Model-Benchmark-for-IMDB-Sentiment-Classification","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahtabRanjbar%2FBERT-Model-Benchmark-for-IMDB-Sentiment-Classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahtabRanjbar%2FBERT-Model-Benchmark-for-IMDB-Sentiment-Classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahtabRanjbar%2FBERT-Model-Benchmark-for-IMDB-Sentiment-Classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahtabRanjbar%2FBERT-Model-Benchmark-for-IMDB-Sentiment-Classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MahtabRanjbar","download_url":"https://codeload.github.com/MahtabRanjbar/BERT-Model-Benchmark-for-IMDB-Sentiment-Classification/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MahtabRanjbar%2FBERT-Model-Benchmark-for-IMDB-Sentiment-Classification/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278924144,"owners_count":26069400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert-fine-tuning","bert-models","classification","llms","nlp"],"created_at":"2024-11-24T10:17:57.337Z","updated_at":"2025-10-08T09:56:19.306Z","avatar_url":"https://github.com/MahtabRanjbar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fine-tuning BERT-based Models for IMDB Sentiment Classification\n\n## Table of Contents\n- [Project Overview](#project-overview)\n- [Data Information](#data-information)\n- [Models Evaluated](#models-evaluated)\n- [Benchmark Results](#benchmark-results)\n- [Analysis](#analysis)\n- [Conclusions](#conclusions)\n- [Getting Started](#getting-started)\n- [Future Work](#future-work)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Project Overview\n\nThis project implements a comprehensive benchmark for fine-tuning various BERT-based pre-trained models on a subset of the IMDB movie review dataset for sentiment classification. We evaluate and compare the performance of these fine-tuned models in terms of accuracy, precision, recall, F1 score, training time, and model size.\n\n## Data Information\n\nWe use a subset of the IMDB dataset for fine-tuning and evaluation:\n\n- **Dataset**: Subset of IMDB Movie Reviews\n- **Task**: Binary Sentiment Classification (Positive/Negative)\n- **Original Size**: 50,000 reviews (25,000 training, 25,000 testing)\n- **Subset Size**: 2,000 reviews (1,000 for training, 1,000 for testing)\n- **Source**: [IMDB Dataset](https://ai.stanford.edu/~amaas/data/sentiment/)\n- **Features**: Text reviews and binary sentiment labels\n\n## Models Evaluated\n\nWe fine-tune the following BERT-based pre-trained models:\n\n- BERT (bert-base-cased)\n- ALBERT (albert-base-v2)\n- RoBERTa (roberta-base)\n- DistilBERT (distilbert-base-uncased)\n\nEach model is fine-tuned on our IMDB subset, adapting its pre-trained knowledge to the specific task of movie review sentiment classification.\n\n## Benchmark Results\n\nResults after fine-tuning on the IMDB subset:\n\n| Model                  | Accuracy | Precision | Recall | F1 Score | Training Time (s) | Parameters  |\n|------------------------|----------|-----------|--------|----------|-------------------|-------------|\n| bert-base-cased        | 0.847    | 0.827     | 0.869  | 0.847    | 262.45            | 108,311,810 |\n| albert-base-v2         | 0.869    | 0.936     | 0.785  | 0.854    | 224.26            | 11,685,122  |\n| roberta-base           | 0.875    | 0.962     | 0.775  | 0.858    | 221.61            | 124,647,170 |\n| distilbert-base-uncased| 0.842    | 0.915     | 0.746  | 0.822    | 113.65            | 66,955,010  |\n\n![Benchmark Results](./results/benchmark_results.png)\n\n## Analysis\n\n1. **Accuracy**: \n   - Fine-tuned RoBERTa performs best (87.5%), followed closely by ALBERT (86.9%).\n   - Fine-tuned BERT and DistilBERT show slightly lower accuracy (84.7% and 84.2% respectively).\n\n2. **Precision and Recall**:\n   - Fine-tuned RoBERTa has the highest precision (96.2%), but lower recall compared to BERT.\n   - Fine-tuned BERT shows the highest recall (86.9%), indicating better performance in identifying positive samples.\n\n3. **F1 Score**:\n   - Fine-tuned RoBERTa leads with an F1 score of 0.858, closely followed by ALBERT (0.854).\n   - Fine-tuned DistilBERT has the lowest F1 score (0.822), suggesting a less balanced performance.\n\n4. **Training Time**:\n   - Fine-tuning DistilBERT is significantly faster (113.65s), about half the time of other models.\n   - Fine-tuning BERT takes the longest (262.45s), while RoBERTa and ALBERT have similar fine-tuning times.\n\n5. **Model Size**:\n   - ALBERT is by far the smallest model (11.7M parameters), making it efficient for deployment.\n   - RoBERTa is the largest (124.6M parameters), followed closely by BERT (108.3M parameters).\n\n6. **Efficiency vs. Performance**:\n   - Fine-tuned ALBERT offers the best balance of performance and efficiency, with high accuracy and the smallest model size.\n   - Fine-tuned RoBERTa provides the highest accuracy but at the cost of model size.\n   - Fine-tuned DistilBERT, while less accurate, offers significant speed advantages and a smaller model size compared to BERT.\n\n## Conclusions\n\nAfter fine-tuning on the IMDB subset:\n\n1. For highest accuracy: Choose RoBERTa\n2. For best balance of performance and efficiency: ALBERT\n3. For fastest inference and smallest model size: DistilBERT\n\nThe choice of model depends on the specific requirements of the application, balancing factors such as accuracy, speed, and resource constraints. These results demonstrate the effectiveness of fine-tuning pre-trained BERT-based models even on a small subset of task-specific data.\n\n## Getting Started\n\n### Installation\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/MahtabRanjbar/BERT-Model-Benchmark-for-IMDB-Sentiment-Classification.git\ncd BERT-Model-Benchmark-for-IMDB-Sentiment-Classification\n```\n\n2. Install dependencies:\n```bash\npip install -r requirements.txt\n```\n\n### Running the Benchmark\n\n1. Configure the fine-tuning and benchmark parameters in `config/config.yaml`\n\n2. Run the benchmark:\n```bash\npython src/main.py\n```\n3. Results will be saved in `benchmark_results.csv` and `benchmark_results.png`.\n\n## Future Work\n\n- Experiment with larger variants of these models (e.g., BERT-large, RoBERTa-large)\n- Increase the subset size to investigate performance on larger datasets\n- Implement cross-validation for more robust results\n- Explore advanced fine-tuning strategies to improve performance\n- Benchmark on other sentiment analysis datasets for comparison\n- Investigate the impact of different preprocessing techniques\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahtabranjbar%2Fbert-model-benchmark-for-imdb-sentiment-classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmahtabranjbar%2Fbert-model-benchmark-for-imdb-sentiment-classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahtabranjbar%2Fbert-model-benchmark-for-imdb-sentiment-classification/lists"}