{"id":28432561,"url":"https://github.com/shubhamhackz/project_b","last_synced_at":"2025-07-01T08:31:59.639Z","repository":{"id":296854585,"uuid":"994739608","full_name":"shubhamhackz/project_b","owner":"shubhamhackz","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-02T18:24:51.000Z","size":1559,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-03T01:21:48.051Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shubhamhackz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-02T12:09:08.000Z","updated_at":"2025-06-02T18:24:55.000Z","dependencies_parsed_at":"2025-06-03T01:35:35.265Z","dependency_job_id":null,"html_url":"https://github.com/shubhamhackz/project_b","commit_stats":null,"previous_names":["shubhamhackz/project_b"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/shubhamhackz/project_b","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubhamhackz%2Fproject_b","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubhamhackz%2Fproject_b/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubhamhackz%2Fproject_b/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubhamhackz%2Fproject_b/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shubhamhackz","download_url":"https://codeload.github.com/shubhamhackz/project_b/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubhamhackz%2Fproject_b/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262927724,"owners_count":23386013,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-05T17:10:15.420Z","updated_at":"2025-07-01T08:31:59.623Z","avatar_url":"https://github.com/shubhamhackz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Production-Grade NER Model\n\n**Single Entry Point Design** - One command for everything: train, test, and deploy!\n\n## 🚀 Quick Start\n\n```bash\n# Full pipeline: Train → Test → Interactive\npython main.py --full-pipeline --real-world\n\n# Individual modes\npython main.py --real-world          # Train only (default)\npython main.py --test               # Test trained model\npython main.py --interactive        # Interactive terminal\n```\n\n## 📋 Commands Overview\n\n| Command | Description |\n|---------|-------------|\n| `python main.py --real-world` | Train robust model with overfitting fixes |\n| `python main.py --test` | Run comprehensive testing suite |\n| `python main.py --interactive` | Start interactive NER interface |\n| `python main.py --full-pipeline --real-world` | Complete workflow |\n\n## 🛠️ Project Structure\n\n```\nproject_b/\n├── main.py                          # 🎯 SINGLE ENTRY POINT\n├── tools/                           # Testing \u0026 interaction utilities\n│   ├── test_model_comprehensive.py  # Comprehensive testing suite\n│   └── interactive_ner.py           # Interactive terminal interface\n├── model.py                         # Model architecture\n├── train.py                         # Training utilities\n├── utils.py                         # Data processing\n├── evaluate.py                      # Evaluation metrics\n└── advanced_real_world_training.py  # Overfitting fixes\n```\n\n## 🎯 Training Options\n\n### Real-World Training (Recommended)\n```bash\npython main.py --real-world --epochs 8 --batch-size 8 --learning-rate 1.5e-5\n```\n\n**Fixes Applied:**\n- ✅ Surface pattern corruption (breaks @ .com memorization)\n- ✅ Realistic targets: EMAIL 85-90%, PHONE 82-88%, PERSON 88-94%\n- ✅ Stronger regularization (weight_decay 0.02, label_smoothing 0.1)\n- ✅ More frequent evaluation (every 100 steps)\n\n### Standard Training\n```bash\npython main.py                      # Basic training\npython main.py --epochs 4           # Custom epochs\npython main.py --batch-size 16      # Custom batch size\n```\n\n## 🧪 Testing\n\n### Comprehensive Testing Suite\n```bash\npython main.py --test\n```\n\n**Tests Include:**\n- ✅ Basic entity recognition (PER, ORG, LOC, MISC)\n- ✅ EMAIL/PHONE detection accuracy\n- ✅ Overfitting detection (corrupted patterns)\n- ✅ Edge cases (empty text, long sequences, special chars)\n- ✅ Performance analysis with 80% success threshold\n\n**Output:** `model_test_report.json` with detailed results\n\n## 🚀 Interactive Mode\n\n```bash\npython main.py --interactive\n```\n\n**Features:**\n- 🎨 Colored entity highlighting\n- ⚡ Real-time processing with timing\n- 📊 Built-in commands: `help`, `examples`, `stats`, `test`\n- 🛡️ Safety checks (requires testing first)\n\n## 🔄 Full Pipeline\n\n```bash\npython main.py --full-pipeline --real-world\n```\n\n**Workflow:**\n1. **Training** - Robust real-world training with overfitting fixes\n2. **Testing** - Comprehensive validation (must pass ≥80%)\n3. **Interactive** - Deploy only if testing passes\n\n## 📊 Expected Performance\n\n| Entity Type | Target F1 | Note |\n|-------------|-----------|------|\n| EMAIL | 85-90% | Not 99%+ (overfitting) |\n| PHONE | 82-88% | Not 99%+ (overfitting) |\n| PERSON | 88-94% | Not 99%+ (overfitting) |\n| OVERALL | 86-92% | Not 98%+ (overfitting) |\n\n## 🔧 Advanced Options\n\n```bash\n# Custom model\npython main.py --model roberta-base --real-world\n\n# More synthetic data\npython main.py --synthetic-count 10000 --real-world\n\n# Quick test run\npython main.py --epochs 2 --batch-size 4\n```\n\n## 📁 File Organization\n\n- **`main.py`** - Central orchestrator, single entry point\n- **`tools/`** - Separate but integrated utilities\n- **`production-ner-model-final/`** - Trained model output\n- **`checkpoints/`** - Training checkpoints\n- **`mlruns/`** - MLflow experiment tracking\n\n## 🎯 Production Deployment\n\n1. **Train:** `python main.py --real-world`\n2. **Validate:** `python main.py --test` (≥80% required)\n3. **Deploy:** `python main.py --interactive`\n\n**Safety:** Interactive mode requires testing completion first!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshubhamhackz%2Fproject_b","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshubhamhackz%2Fproject_b","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshubhamhackz%2Fproject_b/lists"}