{"id":49666716,"url":"https://github.com/prakashy003/mindscope-ai","last_synced_at":"2026-05-06T17:01:30.745Z","repository":{"id":355690995,"uuid":"1087952326","full_name":"prakashy003/MindScope-AI","owner":"prakashy003","description":"An NLP-powered system for detecting and classifying mental health conditions from text using machine learning and transformer models","archived":false,"fork":false,"pushed_at":"2026-05-04T21:26:31.000Z","size":109119,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-04T21:30:13.004Z","etag":null,"topics":["fine-tuning","llms","mental-health","nlp","prompt-engineering","transformers-models"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prakashy003.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-02T01:29:55.000Z","updated_at":"2026-05-04T21:28:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/prakashy003/MindScope-AI","commit_stats":null,"previous_names":["prakashy003/mindscope-ai"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/prakashy003/MindScope-AI","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakashy003%2FMindScope-AI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakashy003%2FMindScope-AI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakashy003%2FMindScope-AI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakashy003%2FMindScope-AI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prakashy003","download_url":"https://codeload.github.com/prakashy003/MindScope-AI/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prakashy003%2FMindScope-AI/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32703532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-06T08:33:17.875Z","status":"ssl_error","status_checked_at":"2026-05-06T08:33:17.221Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fine-tuning","llms","mental-health","nlp","prompt-engineering","transformers-models"],"created_at":"2026-05-06T17:01:29.822Z","updated_at":"2026-05-06T17:01:30.738Z","avatar_url":"https://github.com/prakashy003.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MindScope AI — Mental Health NLP Classification \u0026 LLM Benchmark\n\n## Overview\n**MindScope AI** is an NLP-based system designed to detect and classify mental health conditions from text inputs.  \nUsing the *Sentiment Analysis for Mental Health* dataset from Hugging Face, it explores how linguistic cues reflect emotional and psychological states.\n\nThe project compared multiple modeling paradigms—**TF-IDF + SVM**, **Word2Vec + LSTM**, and **Fine-tuned Transformers (MentalBERT)**—to determine which approach most effectively captures nuanced mental health expressions.\n\n## Model Comparison\n\n- **TF-IDF + SVM**: Achieved a macro F1 score of **0.71**. This traditional approach offered solid baseline performance but struggled with minor classes and complex sentence structures due to a lack of contextual understanding.\n- **Word2Vec + LSTM**: Attained a macro F1 score of **0.69**. While better at understanding sequence patterns, it encountered difficulties with nuanced emotions compared to transformer models.\n- **MentalBERT (Winner)**: Outperformed both baseline models significantly, scoring a test accuracy of **84.2%** and a macro F1 score of **0.84**. This domain-adapted model perfectly matched infrequent targets like \"Personality Disorder\" and showed profound context awareness.\n\n## Deployment\n\nThe successful **MentalBERT** model was deployed via a complete **Flask web application**. Users may now type sentences reflecting different emotional or psychological states into the frontend and get an accurate mental health tag returned in real-time.\nThe interface allows users to quickly test the model with natural language inputs while the backend processes the text using the trained MentalBERT classifier.\n\nWeb Application Interface:\n\n\u003cp align=\"center\"\u003e\n\u003cimg width=\"673\" height=\"904\" alt=\"image\" src=\"https://github.com/user-attachments/assets/9790929e-65c7-48e6-8847-f0ff5d03e136\" /\u003e\n\u003c/p\u003e\nExample of the MindScope AI web interface where users input text and receive mental health classification results.\n\nIn the example above, the user enters the sentence:\n\n\u003e \"I just feel completely empty. I don't see the point in waking up anymore.\"\n\nThe **MentalBERT model** analyzes the text and classifies it as **Suicidal** with a **confidence score of 99.7%**.  \nThe interface displays the predicted label, the model’s confidence score, and a probability breakdown across all supported classes (*Suicidal, Personality Disorder, Stress, Bipolar, Normal, Depression, Anxiety*).\n\nThis visualization helps users quickly understand how the model interprets emotional cues in the input text.\n\n\u003e **Disclaimer:** This tool is for research and demonstration purposes only and does not replace professional medical advice.\n\n---\n\n## LLM Benchmark — Findings\n\nTo push the project further, MentalBERT was benchmarked against **Claude Haiku** and **GPT-4o-mini** across three prompt strategies (zero-shot, few-shot, chain-of-thought) on the same 251-row stratified test set used during training evaluation.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"llm_benchmark/results/macro_f1_comparison.png\" alt=\"Macro F1 Comparison\" width=\"750\"/\u003e\n\u003c/p\u003e\n\n| Model | Strategy | Accuracy | Macro F1 | F1 — Personality Disorder |\n|---|---|---|---|---|\n| **MentalBERT (fine-tuned)** | — | **84.2%** | **0.838** | **0.830** |\n| Claude Haiku | few-shot | 65.3% | 0.634 | 0.409 |\n| GPT-4o-mini | few-shot | 63.3% | 0.610 | 0.375 |\n| Claude Haiku | zero-shot | 59.8% | 0.576 | 0.263 |\n| GPT-4o-mini | CoT | 59.4% | 0.579 | 0.293 |\n| Claude Haiku | CoT | 41.0% | 0.341 | 0.000 |\n\n**Key findings:**\n- MentalBERT leads by **20 macro F1 points** over the best LLM strategy\n- The gap is sharpest on **Personality Disorder** — MentalBERT 0.83 vs best LLM 0.41. General-purpose models struggle most on rare, clinically specific classes\n- **Chain-of-thought backfired on Claude** — reasoning step-by-step dropped macro F1 from 0.576 to 0.341, showing CoT can hurt on subjective classification tasks\n- GPT-4o-mini zero-shot costs **5× less** than Claude Haiku at comparable accuracy — a relevant cost trade-off for production use\n\nFull per-class breakdown and visualizations are in [`llm_benchmark/evaluate.ipynb`](llm_benchmark/evaluate.ipynb).\n\n---\n\n## Why MentalBERT Outperformed Every Approach\n\nAcross all four paradigms tested — traditional ML, word embeddings, Claude, and GPT — MentalBERT consistently ranked first. The reason comes down to one concept: **domain adaptation**.\n\n- **TF-IDF + SVM (F1: 0.71)** treats text as a bag of words with no understanding of order or meaning. It can separate broad categories but collapses on classes like Stress and Personality Disorder where the vocabulary overlaps heavily.\n\n- **Word2Vec + LSTM (F1: 0.69)** learns word relationships and sequence patterns, but its embeddings were trained on general text (Google News), not mental health language. It captures sentence flow but misses domain-specific emotional nuance.\n\n- **Claude Haiku \u0026 GPT-4o-mini (best F1: 0.634)** are powerful general reasoners trained on broad internet data. They perform well on obvious cases — Anxiety and Suicidal — but have never seen the specific linguistic patterns of Reddit mental health communities. Without fine-tuning, they cannot reliably distinguish Stress from Depression, or recognize the indirect language typical of Personality Disorder posts.\n\n- **MentalBERT (F1: 0.838)** starts from BERT pre-trained specifically on mental health corpora, then is fine-tuned directly on this dataset's labeled examples. It has seen the exact style of language users express mental health states in, making it sensitive to subtle cues that general models miss entirely.\n\nThe takeaway: **domain-adapted fine-tuning beats prompt engineering**. For specialized, high-stakes classification tasks, a smaller model trained on the right data consistently outperforms a larger general model given only a prompt.\n\n\u003e **Disclaimer:** This tool is for research and demonstration purposes only and does not replace professional medical advice.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprakashy003%2Fmindscope-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprakashy003%2Fmindscope-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprakashy003%2Fmindscope-ai/lists"}