{"id":25522268,"url":"https://github.com/stuartemiddleton/uos_clpsych","last_synced_at":"2025-07-01T10:02:21.538Z","repository":{"id":43640937,"uuid":"494113148","full_name":"stuartemiddleton/uos_clpsych","owner":"stuartemiddleton","description":"University of Southampton CLPsych 2022 software submission","archived":false,"fork":false,"pushed_at":"2022-08-31T10:00:18.000Z","size":2985,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-21T12:51:30.731Z","etag":null,"topics":["deep-learning","mental-health","natural-language-processing","nlp","text-classification"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stuartemiddleton.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-19T14:47:17.000Z","updated_at":"2024-07-09T15:41:10.000Z","dependencies_parsed_at":"2023-01-16T16:45:17.919Z","dependency_job_id":null,"html_url":"https://github.com/stuartemiddleton/uos_clpsych","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/stuartemiddleton/uos_clpsych","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuartemiddleton%2Fuos_clpsych","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuartemiddleton%2Fuos_clpsych/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuartemiddleton%2Fuos_clpsych/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuartemiddleton%2Fuos_clpsych/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stuartemiddleton","download_url":"https://codeload.github.com/stuartemiddleton/uos_clpsych/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stuartemiddleton%2Fuos_clpsych/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262941510,"owners_count":23388144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","mental-health","natural-language-processing","nlp","text-classification"],"created_at":"2025-02-19T18:19:05.705Z","updated_at":"2025-07-01T10:02:21.504Z","avatar_url":"https://github.com/stuartemiddleton.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# University of Southampton CLPsych 2022 Shared Task Submission\n## Detecting Moments of Change and Suicidal Risks in Longitudinal User Texts Using Multi-task Learning\nThis work (https://aclanthology.org/2022.clpsych-1.19/) describes the classification system proposed for the Computational Linguistics and Clinical Psychology (CLPsych) Shared Task 2022. We propose the use of multitask learning approach with a bidirectional long-short term memory (Bi-LSTM) model for predicting changes in user's mood (Task A) and their suicidal risk level (Task B). The two classification tasks have been solved independently or in an augmented way previously, where the output of one task is leveraged for learning another task, however this work proposes an 'all-in-one' framework that jointly learns the related mental health tasks. Our experimental results (ranked top for task A) suggest that the proposed multi-task framework outperforms the alternative single-task frameworks submitted to the challenge and evaluated via the timeline based and coverage based performance metrics shared by the organisers. We also assess the potential of using various types of feature embedding schemes that could prove useful in initialising the Bi-LSTM model for better multitask learning in the mental health domain.\n\nThis work is part of the UKRI TAS Hub SafeSpacesNLP project https://www.tas.ac.uk/safespacesnlp/ and supported by the Engineering and Physical Sciences Research Council (EP/V00784X/1). If you use any of the resources in this repository, please cite it as:\n\n\n\nTayyaba Azim, Loitongbam Singh, and Stuart Middleton. 2022.__*Detecting Moments of Change and Suicidal Risks in Longitudinal User Texts Using Multi-task Learning*__. In Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology, pages 213–218, Seattle, USA. Association for Computational Linguistics.\n```\n@inproceedings{azim-etal-2022-detecting,\n    title = \"Detecting Moments of Change and Suicidal Risks in Longitudinal User Texts Using Multi-task Learning\",\n    author = \"Azim, Tayyaba  and\n      Singh, Loitongbam  and\n      Middleton, Stuart\",\n    booktitle = \"Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology\",\n    month = July,\n    year = \"2022\",\n    address = \"Seattle, USA\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2022.clpsych-1.19\",\n    pages = \"213--218\",\n    abstract = \"This work describes the classification system proposed for the Computational Linguistics and Clinical Psychology (CLPsych) Shared Task 2022. We propose the use of multitask learning approach with bidirectional long-short term memory (Bi-LSTM) model for predicting changes in user{'}s mood and their suicidal risk level. The two classification tasks have been solved independently or in an augmented way previously, where the output of one task is leveraged for learning another task, however this work proposes an {`}all-in-one{'} framework that jointly learns the related mental health tasks. The experimental results suggest that the proposed multi-task framework outperforms the remaining single-task frameworks submitted to the challenge and evaluated via timeline based and coverage based performance metrics shared by the organisers. We also assess the potential of using various types of feature embedding schemes that could prove useful in initialising the Bi-LSTM model for better multitask learning in the mental health domain.\",\n}\n  ```\n\n## Proposed Framework\n\u003cimg src=\"https://github.com/stuartemiddleton/uos_clpsych/blob/main/image/Pipeline.png\" alt=\"Framework\"\u003e\n\u003cbr\u003e\n\n## License\n\n### Data Set: \nThe CLPsych data set is proprietary and not shared here. Please contact the competition organisers at clpsych2022-organizers@googlegroups.com to get a copy of its distribution.\n### Software: \n - © Copyright University of Southampton, 2022, Highfield, University Road, Southampton SO17 1BJ.\n - Created By : Tayyaba Azim and Gyanendro Loitongbam\n - Created Date : 2022/05/26\n - Project : SafeSpacesNLP (https://www.tas.ac.uk/safespacesnlp/)\n\n## Installation Requirements Under Ubuntu 20.04LTS \n+ The experiments were run on Dell Precision 5820 Tower Workstation with Nvidia Quadro RTX 6000 24 GB GPU using Nvidia CUDA Toolkit 11.7 and Ubunti 20.04 LTS.\n+ Install the following pre-requisite libraries:\n```\npip install -U sentence-transformers\npip install gensim\npip install transformers\npip install tensorflow\npip install keras\n\nPackage                       Version\n----------------------------- --------------------\ngensim                        4.0.1\nkeras                         2.9.0\nsentence-transformers         2.2.0\ntensorflow                    2.9.1\ntransformers                  4.20.1\n\n```\n## Pretrained Models Required\n+ download [fastText embedding vectors](https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M.vec.zip)\n```\ncd \u003cuos_clpsych_dir\u003e\nmkdir dataset\ncd \u003cuos_clpsych_dir\u003e/dataset\nwget -O wiki-news-300d-1M.vec.zip https://dl.fbaipublicfiles.com/fasttext/vectors-english/wiki-news-300d-1M.vec.zip\nunzip wiki-news-300d-1M.vec.zip\n```\n\n## Preparing Data Set\nThe CLPsych 2022 dataset consists of two types of tab delimited CSV files: *timeline_ID*.tsv and *train_users*.json. \n\nEach *timeline_ID*.tsv contains information about user posts (postid, date, label, title, content) in a particular timeline and \nthe filename (*timeline_ID*) is the timeline ID. \n\nEach *train_users*.json consists of { user_ID : { risk_level : Severe|Moderate|Low, timelines : [timeline_ID, ...] }. \n\nWe merged all the files and prepare a single comma delimited CSV file containing both the informations in *timeline_ID*.tsv and *train_users*.json. The following relational table structure shows how we prepared for the single CSV file.\n\n```\n| Timeline_ID | User_ID | User_risk | Content | Post_ID | Post_label |\n```\n\n- *Timeline_ID*: This is the timeline ID of the sequence of posts by a user\n- *User_ID*: The user id of the above timeline.\n- *User_Risk*: It is the user risk label shared with all the timelines of the user above (Severe|Moderate|Low).\n- *Content*: This is the total text merged with title and content for each user post.\n- *Post_ID*: It is the post id.\n- *Post_label*: It is the label of the post to indicate the moment of change (0|IS|ES). IS = switch. ES = escalation.\n\nSave the training and testing sets as *training_dataset.csv* and *testing_dataset.csv* respectively.\n\n## Sentence Embedding Methods\nThere are two types of sentence embedding methods considered for this study (Please refer to the paper for detail explaination):\n+ *sent_emb*: fastText + SBERT \n+ *sent_score_emb*: fastText + SBERT + Task-specific scores\n\n##  Training Models\n```\ncd \u003cuos_clpsych_dir\u003e\nmkdir results\nmkdir model\n\n# train Multitask (use nohup to run in background as training can take a few hours)\nnohup python codes/CLPsych-multitask_text.py --attention_layer 0 --load_classes dataset/training_classes_index.pkl --training_dataset dataset/training_dataset.csv --testing_dataset dataset/testing_dataset.csv --result_dir results/ --save_model 1 \u003e train_stdout.txt \u0026\n\n# train Multitask-attn\nnohup python codes/CLPsych-multitask_text.py --attention_layer 1 --load_classes dataset/training_classes_index.pkl --training_dataset dataset/training_dataset.csv --testing_dataset dataset/testing_dataset.csv --result_dir results/ --save_model 1 \u003e train_stdout.txt \u0026\n\n# train Multitask-score (use nohup to run in background as training can take a few days)\nnohup python codes/CLPsych-multitask_text-score.py --attention_layer 0 --load_classes dataset/training_classes_index.pkl --training_dataset dataset/training_dataset.csv --testing_dataset dataset/testing_dataset.csv --result_dir results/ --save_model 1 \u003e train_stdout.txt \u0026\n\n# train Multitask-attn-score\nnohup python codes/CLPsych-multitask_text-score.py --attention_layer 1 --load_classes dataset/training_classes_index.pkl --training_dataset dataset/training_dataset.csv --testing_dataset dataset/testing_dataset.csv --result_dir results/ --save_model 1 \u003e train_stdout.txt \u0026\n```\n+ *attention_layer*: Flag to define whether the model to be trained is with (1) or without (0) attention layer.\n+ *load_classes*: Location to load or save the pickle of training class indices.\n+ *training_dataset*: Location of the training dataset file.\n+ *testing_dataset*: Location of the testing dataset file.\n+ *result_dir*: Location to save the model predicted results.\n+ *save_model*: Flag to save model yes (1) or not (0).\n\n## Testing Models\n```\n# test Multitask\npython codes/CLPsych-multitask_text_testing.py --attention_layer 0 --load_classes dataset/training_classes_index.pkl --testing_dataset dataset/testing_dataset.csv --result_dir results/\n\n# test Multitask-attn\npython codes/CLPsych-multitask_text_testing.py --attention_layer 1 --load_classes dataset/training_classes_index.pkl --testing_dataset dataset/testing_dataset.csv --result_dir results/\n\n# test Multitask-score\npython codes/CLPsych-multitask_text_testing_score.py --attention_layer 0 --load_classes dataset/training_classes_index.pkl --testing_dataset dataset/testing_dataset.csv --result_dir results/\n\n# test Multitask-attn-score\npython codes/CLPsych-multitask_text_testing_score.py --attention_layer 1 --load_classes dataset/training_classes_index.pkl --testing_dataset dataset/testing_dataset.csv --result_dir results/\n```\n+ *Multitask*: model using *sent_emb* \n+ *Multitask-score*: model using *sent_score_emb* \n+ *Multitask-attn*: model with attention layer using *sent_emb*\n+ *Multitask-attn-score*: model with attention layer using *sent_score_emb*. \n\n## Evaluating models on Shared Task 2022 (run by our team locally)\n```\n# eval script for task a (using trainingset here but you should provide a validation or test set)\npython codes/evaluate.py results/training_dataset_multitask-taska-with_attention.csv no\npython codes/evaluate.py results/training_dataset_multitask-taska-without_attention.csv no\npython codes/evaluate.py results/training_dataset_multitask-score-taska-without_attention.csv no\npython codes/evaluate.py results/training_dataset_multitask-score-taska-with_attention.csv no\n\n# eval script for task b (using trainingset here but you should provide a validation or test set)\npython codes/evaluate.py results/training_dataset_multitask-taskb-with_attention.csv yes\npython codes/evaluate.py results/training_dataset_multitask-taskb-without_attention.csv yes\npython codes/evaluate.py results/training_dataset_multitask-score-taskb-without_attention.csv yes\npython codes/evaluate.py results/training_dataset_multitask-score-taskb-with_attention.csv yes\n\n```\n**Task A: Moments of Change**\n| Model | Precision | Recall | F1 |\n| ----- | --------- | ------ | -- |\n| *Multitask*\t| 0.582\t| 0.717\t| 0.629\t| \n| *Multitask-attn*\t| 0.663\t| 0.697\t| 0.676\t| \n| *Multitask-score*\t| 0.680\t| 0.760\t| 0.713\t| \n| *Multitask-attn-score*\t| 0.674\t| 0.800\t| 0.724\t| \n\n**Task B: Suicidal Risk Levels**\n| Model | Precision | Recall | F1 |\n| ----- | --------- | ------ | -- |\n| *Multitask*\t| 0.352\t| 0.327\t| 0.335\t| \n| *Multitask-attn*\t|  0.408\t| 0.378\t| 0.388\t| \n| *Multitask-score*\t|  0.355\t| 0.331\t| 0.334\t| \n| *Multitask-attn-score*\t| 0.415\t| 0.397\t| 0.382\t| \n\n## Evaluating models on Shared Task 2022 Test Set Results (run by CLPsych-2022 organisers on our behalf using hidden test labels)\n**Task A: Moments of Change**\n| Model | Precision | Recall | F1 |\n| ----- | --------- | ------ | -- |\n| *Multitask*\t| 0.680\t| 0.579\t| 0.649\t| \n| *Multitask-score*\t| 0.677\t| 0.595\t| 0.625\t| \n| *Multitask-attn-score*\t| 0.680\t| 0.579\t| 0.607\t| \n\n\n**Task B: Suicidal Risk Levels**\n| Model | Precision | Recall | F1 |\n| ----- | --------- | ------ | -- |\n| *Multitask-attn-score*\t| 0.618\t| 0.427\t| 0.451\t| \n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstuartemiddleton%2Fuos_clpsych","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstuartemiddleton%2Fuos_clpsych","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstuartemiddleton%2Fuos_clpsych/lists"}