{"id":27140356,"url":"https://github.com/ozcanmiraay/bart-text-summarization","last_synced_at":"2026-05-15T20:04:13.433Z","repository":{"id":285272341,"uuid":"957576990","full_name":"ozcanmiraay/bart-text-summarization","owner":"ozcanmiraay","description":"End-to-end text summarization pipeline using BART, MLflow, and FastAPI","archived":false,"fork":false,"pushed_at":"2025-04-25T19:24:57.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-25T20:19:10.165Z","etag":null,"topics":["bart","deep-learning","fastapi","huggingface","mlflow","mps","nlp","text-summarization","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ozcanmiraay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-30T17:51:33.000Z","updated_at":"2025-04-25T19:25:00.000Z","dependencies_parsed_at":"2025-03-30T18:42:28.471Z","dependency_job_id":null,"html_url":"https://github.com/ozcanmiraay/bart-text-summarization","commit_stats":null,"previous_names":["ozcanmiraay/bart-text-summarization"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ozcanmiraay/bart-text-summarization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozcanmiraay%2Fbart-text-summarization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozcanmiraay%2Fbart-text-summarization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozcanmiraay%2Fbart-text-summarization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozcanmiraay%2Fbart-text-summarization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ozcanmiraay","download_url":"https://codeload.github.com/ozcanmiraay/bart-text-summarization/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozcanmiraay%2Fbart-text-summarization/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266761419,"owners_count":23980297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bart","deep-learning","fastapi","huggingface","mlflow","mps","nlp","text-summarization","transformers"],"created_at":"2025-04-08T05:51:52.793Z","updated_at":"2026-05-15T20:04:08.403Z","avatar_url":"https://github.com/ozcanmiraay.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🧠 Text Summarization: Model Benchmarking with BART, GPT-2 \u0026 MLflow\n\nThis project demonstrates a lightweight **summarization benchmarking pipeline** using [Hugging Face Transformers](https://github.com/huggingface/transformers), evaluated on the [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset.\n\nIt is built for **reproducible experimentation**, with:\n\n- Short training runs on pre-trained models like `facebook/bart-base` and `gpt2`\n- Consistent evaluation using ROUGE metrics and generation length\n- **MLflow logging** for tracking hyperparameters, metrics, and sample outputs\n- A modular CLI-based script: `src/fine_tune.py` (*used for benchmarking, not fine-tuning*)\n\n\u003e 💡 This project is designed to explore how different model types and configurations affect summarization quality — not to perform full-scale fine-tuning.\n\n---\n\n### 🎥 Demo Walkthrough\n\nWant to see the project in action without setting it up?\n\n✅ **Watch the full demo here:**  \n[📂 Google Drive Folder – Project Demo](https://drive.google.com/drive/folders/13d9DSMaWFTYVKHugG9ifXBiRbJEaF99F?usp=sharing)\n\nIncludes:\n- MLflow walkthrough and run comparison  \n- Analysis outputs and visualizations  \n- Final model usage  \n- Live drift detection in action via FastAPI\n\n---\n\n### 🧱 Project Structure\n\n```\ntext_summarization_project/\n├── checkpoints/               # Model checkpoints from small-scale runs\n├── checkpoints_final/         # Final model checkpoint from best config\n├── deployment/model/          # Saved tokenizer and model for inference\n├── fine_tune_analysis/        # Analysis outputs (CSVs, plots)\n├── mlruns/                    # MLflow run logs (auto-created)\n├── src/\n│   ├── fine_tune.py           # CLI-based benchmarking script\n│   ├── run_sweep.py           # Loop over all model configs\n│   ├── final_train.py         # Run best config at larger scale\n│   ├── analysis_scripts/\n│   │   ├── extract_fine_tune_results_mlflow.py\n│   │   └── fine_tune_analysis.py\n│   └── mlops_demo/\n│       ├── inference_api.py   # FastAPI app to serve model\n│       ├── demo_client.py     # Sends sample requests to server\n│       ├── drift_monitor.py   # Drift detection logic\n│       └── drift_analyzer.py  # Visualizes drift logs\n├── requirements.txt\n└── README.md\n```\n---\n\n### ⚙️ Step 1: Setup Instructions\n\n1. **Clone the repository**:\n\n   ```bash\n   git clone https://github.com/yourusername/text_summarization_project.git\n   cd text_summarization_project\n   ```\nCreate and activate a virtual environment:\n\n```bash\npython -m venv venv\nsource venv/bin/activate  # On macOS/Linux\n# Or: venv\\Scripts\\activate  # On Windows\n```\n\nInstall dependencies:\n\n```bash\npip install -r requirements.txt\n```\n(macOS only): Add support for Apple Silicon GPUs:\n\n```python\nimport os\nos.environ[\"PYTORCH_MPS_HIGH_WATERMARK_RATIO\"] = \"0.0\"\n```\n\n🚀 Run a Benchmarking Trial\nUse the CLI-based script at src/fine_tune.py to evaluate a summarization model with a chosen config.\n\n▶️ Example: Run BART with small batch and short input/output lengths\n```bash\npython src/fine_tune.py \\\n  --model_name_or_path facebook/bart-base \\\n  --epochs 1 \\\n  --batch_size 2 \\\n  --max_input_length 256 \\\n  --max_target_length 64\n```\n\n▶️ Example: Run GPT-2 as a causal LM\n```bash\npython src/fine_tune.py \\\n  --model_name_or_path gpt2 \\\n  --epochs 1 \\\n  --batch_size 2 \\\n  --max_input_length 256 \\\n  --max_target_length 64 \\\n  --use_causal_lm\n```\n\nThis will:\n\n- Run a short training + evaluation loop over ~5000 train and ~250 val examples\n- Log losses, ROUGE scores, and runtime metrics to MLflow\n- Save a few sample predictions as .txt files\n- Track hyperparameters and outcomes for comparison across runs\n\n📂 Output Summary\nEach run logs:\n\n- 📉 Training \u0026 evaluation metrics (loss, ROUGE, speed)\n- 📑 Token length stats for predictions and references\n- 📝 Sample outputs: article, reference summary, model prediction\n- 🧾 Run-specific artifacts via MLflow (including config and example text files)\n- 📈 View MLflow Logs\n\n---\n\n### 🧪 Step 2: Reproduce All Experiments\nTo run the exact 10 benchmarking experiments used in this project, use the src/run_sweep.py script. This script loops through 10 combinations of:\n\n- Model type (facebook/bart-base, gpt2)\n- Epochs (1 or 2)\n- Batch size (2 or 4)\n- Input length (256 or 384)\n- Target length (64 or 128)\n- Model family (Seq2Seq or Causal LM)\n\nEach experiment is run sequentially and logged via MLflow under the same \"local-file\" experiment name.\n\n▶️ Run All Benchmarking Sweeps\n```bash\npython src/run_sweep.py\n```\n\nThis will:\n\n- Run all 10 model configuration experiments defined in the sweep list\n- Automatically handle BART vs. GPT-2 configuration logic\n- Log all metrics, ROUGE scores, and samples to MLflow\n- Sleep for 5 seconds between runs to ensure system stability\n\n⚠️ Make sure you've already set up your environment and installed requirements before running the sweep.\n\nAfter it's done, view all runs in one place using the MLflow UI:\n\n```bash\nmlflow ui\n```\nThen go to: http://127.0.0.1:5000 and browse the experiment \"local-file\".\n\n---\n\n### 📊 Step 3: Analyze Results\n\nOnce all experiments have been logged via MLflow, you can extract and analyze the benchmarking results using the following two scripts under `src/analysis_scripts/`.\n\n---\n\n#### 📄 `extract_fine_tune_results_mlflow.py`\n\nThis script exports all run metadata and final metrics into a single, clean `.csv` file for further analysis or visualization.\n\nIt captures:\n- Model config parameters (e.g., model type, batch size, input/output lengths)\n- Final ROUGE scores, eval loss, and training speed\n- Run info such as status and start time\n\n✅ **Run it after completing all experiments**:\n\n```bash\npython src/analysis_scripts/extract_fine_tune_results_mlflow.py\n```\n\nThis will save the file to:\n\n```\nfine_tune_analysis/mlflow_all_model_runs.csv\n```\n\n---\n\n#### 📈 `fine_tune_analysis.py`\n\nThis script loads all MLflow runs directly, filters the key metrics and parameters, and produces:\n\n- ✅ Leaderboards for best/worst runs\n- 📊 A runtime vs. ROUGE-1 scatter plot\n- 📊 A ROUGE-1/2/L bar chart (best run per model)\n- 📊 A parallel coordinates plot showing hyperparameter impact\n- 📁 A `summary.csv` file for downstream Excel / pandas analysis\n\n✅ **Run it anytime after experiments are logged**:\n\n```bash\npython src/analysis_scripts/fine_tune_analysis.py\n```\n\nThis will output:\n- 3 figures saved to: `fine_tune_analysis/`\n- CSV summary: `fine_tune_analysis/summary.csv`\n\n\u003e 📍 Make sure MLflow still points to the same `mlruns/` directory.\n\n---\n\n### 🏁 Step 4: Run the Best Model at Scale\n\nAfter completing the sweep and analysis steps, we identified the best-performing configuration (based on ROUGE-1 and eval loss) and ran it on a **larger dataset slice** for a more robust final evaluation.\n\nThis was done using the `src/final_train.py` script.\n\n---\n\n#### 📄 `final_train.py`\n\nThis script reruns the top configuration from the sweep:\n- `facebook/bart-base`\n- `1` epoch\n- `batch_size=2`\n- `max_input_length=384`, `max_target_length=64`\n\n...but scales up the dataset:\n- `train`: 25,000 examples  \n- `validation`: 1,250 examples  \n- `test`: 1,250 examples\n\nIt performs:\n- ✅ Full training on the larger training split  \n- 📊 ROUGE-based evaluation on both validation and test sets  \n- 📝 Logging of metrics, lengths, and prediction examples to MLflow  \n- 💾 Saving of the final model and tokenizer to `deployment/model/` for downstream use\n\n---\n\n#### ▶️ Run the final training script\n\n```bash\npython src/final_train.py\n```\n\nThis will:\n- Log a new MLflow run named `bart-base_final_benchmark_config`\n- Store the final model under: `deployment/model/`\n- Log validation and test ROUGE scores to MLflow\n- Upload sample predictions to MLflow as artifacts\n\n\u003e 📦 The saved model can now be reused for inference or integrated into a downstream application or API.\n\n---\n\n### ⚙️ Step 5: MLOps Demo – Inference \u0026 Drift Monitoring\n\nThis project includes a complete, lightweight **MLOps simulation** using FastAPI and basic drift monitoring heuristics.\n\n---\n\n#### 🛰️ Start the Inference Server (Terminal 1)\n\nThis will load the saved model from `deployment/model/` and expose a `/summarize` endpoint.\n\n```bash\nuvicorn src.mlops_demo.inference_api:app --reload --port 8000\n```\n\nThe server will:\n- Tokenize and summarize incoming input\n- Log summary token length\n- Call drift monitors on:\n  - Input entropy \u0026 readability\n  - Summary length deviation\n  - Embedding-based cosine drift\n\nDrift logs are saved to:\n\n```\nmlops_demo/drift_logs.log\nmlops_demo/alerts.json\n```\n\n---\n\n#### 🤖 Run the Client Simulator (Terminal 2)\n\nThis script sends both real CNN/DailyMail articles and synthetic \"drift-triggering\" inputs to the server.\n\n```bash\npython src/mlops_demo/demo_client.py\n```\n\nThe script simulates:\n- Normal requests from the dataset (real-world cases)\n- Drift cases including:\n  - Short or very long inputs\n  - Low entropy (repeating tokens)\n  - High entropy (gibberish)\n  - Embedding-based novelty\n\nEach result includes latency, token count, and a truncated summary.\n\n---\n\n#### 📉 Drift Logging \u0026 Monitoring\n\nThe following drift types are monitored in `drift_monitor.py`:\n\n- **Output Length Drift**  \n  Triggers when average summary length deviates from baseline (56 tokens ±10)\n- **Input Entropy Drift**  \n  Flags abnormally repetitive or highly chaotic input\n- **Embedding Drift**  \n  Computes cosine distance to a reference embedding baseline\n\nDrift alerts are logged to:\n- `mlops_demo/drift_logs.log` (for audit/debug)\n- `mlops_demo/alerts.json` (for alert storage)\n\n---\n\n#### 📊 Visualize Drift Over Time\n\nUse the following script to generate 4 time-series plots based on the logs:\n\n```bash\npython src/mlops_demo/drift_analyzer.py\n```\n\nIt will show:\n- Input length over time  \n- Input entropy and entropy-based drift  \n- Summary length and output drift  \n- Embedding cosine distance vs. threshold  \n\n---\n\n\u003e 🖥️ **Note:** You must run the server (`uvicorn ...`) and the client (`python demo_client.py`) in **separate terminals** at the same time.\n\n---\n\n### 🚀 Next Steps\n\n- ✅ Add Docker support for containerized deployment\n- ✅ Integrate real-time metrics dashboard for live monitoring\n- 🧪 Experiment with larger models (e.g., `bart-large`, `t5-base`)\n- 🧠 Incorporate LoRA or quantization for efficient fine-tuning\n- 🌐 Deploy API to a cloud platform (e.g., Hugging Face Spaces, Render)\n\n\u003e Contributions and ideas are welcome!\n\n---\n### 👋 Contact\n\nBuilt by **Miray Özcan**  \n📧 `miray@uni.minerva.edu`  \n🌐 [linkedin.com/in/mirayozcan](https://linkedin.com/in/mirayozcan)\n\n\u003e If you found this useful or want to collaborate, feel free to reach out!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fozcanmiraay%2Fbart-text-summarization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fozcanmiraay%2Fbart-text-summarization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fozcanmiraay%2Fbart-text-summarization/lists"}