{"id":13525728,"url":"https://github.com/SALT-NLP/FLANG","last_synced_at":"2025-04-01T05:32:13.830Z","repository":{"id":68595384,"uuid":"421259354","full_name":"SALT-NLP/FLANG","owner":"SALT-NLP","description":"When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain","archived":false,"fork":false,"pushed_at":"2025-02-11T23:23:41.000Z","size":1403,"stargazers_count":46,"open_issues_count":3,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-12T00:24:57.882Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SALT-NLP.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-26T02:57:09.000Z","updated_at":"2025-01-27T10:52:00.000Z","dependencies_parsed_at":"2024-11-02T10:31:08.687Z","dependency_job_id":"caaf0328-3306-4968-8390-640969c425f8","html_url":"https://github.com/SALT-NLP/FLANG","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SALT-NLP%2FFLANG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SALT-NLP%2FFLANG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SALT-NLP%2FFLANG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SALT-NLP%2FFLANG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SALT-NLP","download_url":"https://codeload.github.com/SALT-NLP/FLANG/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246591651,"owners_count":20801983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T06:01:21.514Z","updated_at":"2025-04-01T05:32:12.959Z","avatar_url":"https://github.com/SALT-NLP.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话","Python","📏 评测基准"],"sub_categories":["大语言对话模型及数据","🧩 领域模型"],"readme":"# When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain\n\n## Abstract\n\u003cp align=\"justify\"\u003e\nPre-trained language models have shown impressive performance on a variety of tasks and domains. Previous research on financial language models usually employs a generic training scheme to train standard model architectures, without completely leveraging the richness of the financial data. We propose a novel domain specific Financial LANGuage model (FLANG) which uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective. Additionally, the evaluation benchmarks in the field have been limited. To this end, we contribute the Financial Language Understanding Evaluation (FLUE), an open-source comprehensive suite of benchmarks for the financial domain. These include new benchmarks across 5 NLP tasks in financial domain as well as common benchmarks used in the previous research. Experiments on these benchmarks suggest that our model outperforms those in prior literature on a variety of NLP tasks. \n\u003c/p\u003e\n\n\n## FLANG-ELECTRA Architecture\n![Architecture of our model. We use finance specific datasets and general English datasets (Wikpedia and BooksCorpus) for training the model. We follow the training strategy of ELECTRA with span boundary task which first predicts masked tokens using language model and then uses a discriminator to assess if a token is original or replaced. The generator and discriminator are trained end-to-end, and both words and phrases from financial vocabulary are used for masking. The final discriminator is then fine-tuned on individual tasks on our contributed benchmark suite, Financial Language Understanding Evaluation (FLUE). Note that our method is not specific to ELECTRA and can be generalized to other models.](/images/flang.jpg)\n\u003csub\u003eArchitecture of our model. We use finance specific datasets and general English datasets (Wikpedia and BooksCorpus) for training the model. We follow the training strategy of ELECTRA with span boundary task which first predicts masked tokens using language model and then uses a discriminator to assess if a token is original or replaced. The generator and discriminator are trained end-to-end, and both words and phrases from financial vocabulary are used for masking. The final discriminator is then fine-tuned on individual tasks on our contributed benchmark suite, Financial Language Understanding Evaluation (FLUE). Note that our method is not specific to ELECTRA and can be generalized to other models.\u003c/sub\u003e\n\n\n## FLUE: Financial Language Understanding Evaluation\nFLUE (Financial Language Understanding Evaluation) is a comprehensive and heterogeneous benchmark that has been built from 5 diverse financial domain specific datasets.\n\n\n| Name       | Task                             | Source                  | Dataset Size |\n|------------|----------------------------------|-------------------------|--------------|\n| FPB        | Financial Sentiment Analysis     | Malo et al. 2014b       | 4,845        |\n| FiQA SA    | Financial Sentiment Analysis     | FiQA 2018               | 1,173        |\n| Headline   | News Headline Classification     | Sinha and Khandait 2020 | 11,412       |\n| NER        | Named Entity Recognition         | Alvarado et al. 2015    | 1,466        |\n| FinSBD3    | Structure Boundary Detection     | FinSBD3 (FinWeb-2021)   | 756          |\n| FiQA QA    | Question Answering               | FiQA 2018               | 6,640        |\n\n### Performance of FLANG Architecture of FLUE datasets\n\n| Model/Metric        | FPB      | FiQA SA | Headline | NER  | FinSBD3 | FiQA QA |\n| ------------------- | -------- | ------- | -------- | ---- | ------- | ------- |\n|                     | Accuracy | MSE     | Mean F-1 | F-1  | F-1     | nDCG    |\n| BERT-base           | 0.856    | 0.073   | 0.967    | 0.79 | 0.95    | 0.46    |\n| FinBERT             | 0.872    | 0.07    | 0.968    | 0.8  | 0.89    | 0.42    |\n| FLANG-BERT(ours)    | 0.912    | 0.054   | 0.972    | 0.83 | 0.96    | 0.51    |\n| ELECTRA             | 0.881    | 0.066   | 0.966    | 0.78 | 0.94    | 0.52    |\n| FLANG-ELECTRA(ours) | 0.919    | 0.034   | 0.98     | 0.82 | 0.97    | 0.55    |\n\n### Financial Sentiment Analysis\n1. Financial PhraseBank (Classification)\n    * Data: [Financial PhraseBank](https://huggingface.co/datasets/financial_phrasebank)\n    * Cite: ```Malo, Pekka, et al. \"Good debt or bad debt: Detecting semantic orientations in economic texts.\" Journal of the Association for Information Science and Technology 65.4 (2014): 782-796.```\n2. FiQA 2018 Task-1 (Regression)\n    * Data and Ref: [FiQA 2018](https://huggingface.co/datasets/SALT-NLP/FLUE-FiQA)\n    * Cite: ```Maia, Macedo \u0026 Handschuh, Siegfried \u0026 Freitas, Andre \u0026 Davis, Brian \u0026 McDermott, Ross \u0026 Zarrouk, Manel \u0026 Balahur, Alexandra. (2018). WWW'18 Open Challenge: Financial Opinion Mining and Question Answering. WWW '18: Companion Proceedings of the The Web Conference 2018. 1941-1942. 10.1145/3184558.3192301.``` \n\n### News Headline Classification\n * Data: [Gold Commodity News and Dimensions](https://www.kaggle.com/datasets/daittan/gold-commodity-news-and-dimensions)\n * Cite: ```Sinha, A., \u0026 Khandait, T. (2021, April). Impact of News on the Commodity Market: Dataset and Results. In Future of Information and Communication Conference (pp. 589-601). Springer, Cham.```\n\n### Named Entity Recognition\n * Data: [NER Data on Loan Agreement](https://paperswithcode.com/dataset/fin)\n * Cite: ```Alvarado, J. C. S., Verspoor, K., \u0026 Baldwin, T. (2015, December). Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Technology Association Workshop 2015 (pp. 84-90).```\n\n### Structure Boundary Detection\n * Data: [FinSBD3](https://sites.google.com/nlg.csie.ntu.edu.tw/finweb2021/shared-task-finsbd-3)\n * Cite: ```Willy Au, Abderrahim Ait-Azzi, and Juyeon Kang. 2021. FinSBD-2021: The 3rd Shared Task on Structure Boundary Detection in Unstructured Text in the Financial Domain. In Companion Proceedings of the Web Conference 2021 (WWW '21). Association for Computing Machinery, New York, NY, USA, 276–279. https://doi.org/10.1145/3442442.3451378```\n\n### Question Answering\n * Data and Ref: [FiQA 2018](https://huggingface.co/datasets/SALT-NLP/FLUE-FiQA)\n * Cite: ```Maia, Macedo \u0026 Handschuh, Siegfried \u0026 Freitas, Andre \u0026 Davis, Brian \u0026 McDermott, Ross \u0026 Zarrouk, Manel \u0026 Balahur, Alexandra. (2018). WWW'18 Open Challenge: Financial Opinion Mining and Question Answering. WWW '18: Companion Proceedings of the The Web Conference 2018. 1941-1942. 10.1145/3184558.3192301.```\n\n## Leaderboard\nComing soon!\n\n## Citation\nPlease cite the model with the following citation:\n```bibtex\n@INPROCEEDINGS{shah-etal-2022-flang,\n    author = {Shah, Raj Sanjay  and\n      Chawla, Kunal and\n      Eidnani, Dheeraj and\n      Shah, Agam and\n      Du, Wendi and\n      Chava, Sudheer and\n      Raman, Natraj and\n      Smiley, Charese and\n      Chen, Jiaao and\n      Yang, Diyi },\n    title = {When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain},\n    booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)},\n    year = {2022},\n    publisher = {Association for Computational Linguistics}\n}\n```\n\n## Contact information\nPlease contact Raj Sanjay Shah (rajsanjayshah[at]gatech[dot]edu) or Sudheer Chava (schava6[at]gatech[dot]edu) or Diyi Yang (diyiy[at]stanford[dot]edu) about any issues and questions.\n\n\n## Steps to use the code\n\n1. Clone the Repo\n2. cd into the repo in your terminal\n\n## Dependencies\nInstall dependencies with the following command\npip install -r requirements.txt\n\n### Raw data\n\ntokens.npy contains the tokens for financial vocabulary in a numpy array format.\n\n\n\nTo train FLANG-BERT, run\n\n    python train_FLANG_BERT.py\n\nTo train FLANG-ELECTRA, run\n\n    python train_FLANG_ELECTRA.py\n    \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSALT-NLP%2FFLANG","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSALT-NLP%2FFLANG","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSALT-NLP%2FFLANG/lists"}