{"id":25819002,"url":"https://github.com/pngo1997/text-summarization-generation-using-llms","last_synced_at":"2026-04-17T03:35:14.588Z","repository":{"id":276214127,"uuid":"928596578","full_name":"pngo1997/Text-Summarization-Generation-using-LLMs","owner":"pngo1997","description":"Explores text summarization using multiple approaches.","archived":false,"fork":false,"pushed_at":"2025-02-06T22:48:05.000Z","size":13024,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-28T14:13:29.345Z","etag":null,"topics":["bert-score","encoder-decoder-model","fine-tuning","google-t5","large-language-models","llm","mistral-7b","nlp","perplexity","prompt-engineering","python","state-of-the-art-models","text-generation"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pngo1997.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-06T22:28:42.000Z","updated_at":"2025-02-06T23:03:16.000Z","dependencies_parsed_at":"2025-02-06T23:38:00.697Z","dependency_job_id":null,"html_url":"https://github.com/pngo1997/Text-Summarization-Generation-using-LLMs","commit_stats":null,"previous_names":["pngo1997/text-summarization-generation-using-llms"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pngo1997/Text-Summarization-Generation-using-LLMs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pngo1997%2FText-Summarization-Generation-using-LLMs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pngo1997%2FText-Summarization-Generation-using-LLMs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pngo1997%2FText-Summarization-Generation-using-LLMs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pngo1997%2FText-Summarization-Generation-using-LLMs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pngo1997","download_url":"https://codeload.github.com/pngo1997/Text-Summarization-Generation-using-LLMs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pngo1997%2FText-Summarization-Generation-using-LLMs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31913769,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-16T18:22:33.417Z","status":"online","status_checked_at":"2026-04-17T02:00:06.879Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert-score","encoder-decoder-model","fine-tuning","google-t5","large-language-models","llm","mistral-7b","nlp","perplexity","prompt-engineering","python","state-of-the-art-models","text-generation"],"created_at":"2025-02-28T08:14:21.744Z","updated_at":"2026-04-17T03:35:14.565Z","avatar_url":"https://github.com/pngo1997.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📝 Text Summarization using Encoder-Decoder Models \u0026 Large Language Models  \n\n## 📜 Overview  \nThis project explores **text summarization** using multiple approaches:  \n\n1. **Pre-trained Encoder-Decoder Models** (T5) for text summarization.  \n2. **Fine-tuning a T5 model** on a new dataset for improved summarization.  \n3. **Prompt-based Summarization using Large Language Models (LLMs)** for flexible and adaptive text generation.  \n\n📌 **Key Concepts Covered**:  \n- **Encoder-Decoder Models for Summarization**  \n- **Model Evaluation Metrics**: ROUGE, Perplexity, BERTScore  \n- **Fine-Tuning Transformer Models with Hugging Face**  \n- **Prompt Engineering for LLMs**\n  \n## 🚀 1️⃣ Implementation Details  \n\n### **Summarization using Pre-trained Models**  \n✅ **Models Used**:  \n- `\"google-t5/t5-small\"` (general T5 model)  \n- `\"ubikpt/t5-small-finetuned-cnn\"` (fine-tuned on CNN-DailyMail)  \n\n✅ **Dataset**:  \nThis project uses the **CNN-DailyMail Dataset**, available on Hugging Face:  \n🔗 [CNN-DailyMail Dataset](https://huggingface.co/datasets/abisee/cnn_dailymail)  \n\n✅ **Processing Steps**:  \n- Use the **article** column as input text and the **highlights** column as the reference summary.  \n- Tokenize using T5’s tokenizer and ensure **PyTorch tensor format**.  \n\n✅ **Evaluation Metrics**:  \n- **ROUGE-1 \u0026 ROUGE-2** (Lexical overlap)  \n- **Perplexity** (Fluency measure, using GPT-2)  \n- **BERTScore** (Semantic similarity)  \n\n📌 **Comparison**: Generated summaries from both models are compared against ground truth summaries.  \n\n### **Fine-Tuning a T5 Model for Summarization**  \n\n✅ **Model Training Steps**:  \n- Use `\"google-t5/t5-small\"`  \n- **Preprocess data** (clean text, tokenize, create PyTorch DatasetDict).  \n- **Train for 3 epochs** using Hugging Face's `Seq2SeqTrainer`.  \n- **Evaluate on test set** and compare results.  \n\n✅ **Metrics Computed**:  \n- **ROUGE scores**  \n- **Perplexity**  \n- **BERTScore**  \n\n📌 **Comparison**: Performance of the **fine-tuned T5 model** is compared with the pre-trained T5 model.  \n\n### **Prompt-Based Summarization using Large Language Models**  \n✅ **LLMs Used**:  \n🔗 [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)\n\n✅ **Prompt Engineering**:  \n- **Craft multiple prompts** for generating summaries.  \n- **Experiment with few-shot learning** (providing examples in prompts).  \n- **Evaluate how prompt changes affect summary quality**.  \n\n📌 **Comparison**:  \n- Analyze summaries from different LLMs \u0026 prompts.  \n- Compare with fine-tuned T5 model outputs.  \n\n## 📊 2️⃣ Results \u0026 Observations  \n\n### **Performance Metrics**  \n![Metrics](https://github.com/pngo1997/Images/blob/main/HF.png)  \n\n📌 **Key Insights**:  \n- Overall, fine-tuned model performs better on all metrics, which is what expected since fine-tuning would help the model generate more accurate and relevant summaries. The higher precision and F1 scores suggest an improvement in the relevance and completeness of generated content. The drop in perplexity also indicates that the fine-tuned model is more confident in its predictions. \nAdditionally. Optuna was applied to try with different parameters and came to conclusion that the simpler the better. Set max_new_tokens and max_targetLength to 64 because the maximum word count of 'description' per observation is 28 tokens –do not want the model to over generate. \n- **Fine-tuning improves summary quality** (higher ROUGE \u0026 lower Perplexity).  \n- **LLMs generate better summaries than pre-trained models** when prompted well.  \n- **Prompt design significantly affects summary coherence \u0026 informativeness**.  \n\n## 📌 Summary  \n✅ Compared pre-trained and fine-tuned T5 summarization models.\n\n✅ Fine-tuned T5 on a new dataset for improved results.\n\n✅ Explored prompt-based summarization using SOTA LLMs.\n\n✅ Analyzed impact of different evaluation metrics.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpngo1997%2Ftext-summarization-generation-using-llms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpngo1997%2Ftext-summarization-generation-using-llms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpngo1997%2Ftext-summarization-generation-using-llms/lists"}