{"id":23237025,"url":"https://github.com/nicolay-r/reasoning-for-sentiment-analysis-framework","last_synced_at":"2025-08-19T23:31:16.180Z","repository":{"id":227264588,"uuid":"762215378","full_name":"nicolay-r/Reasoning-for-Sentiment-Analysis-Framework","owner":"nicolay-r","description":"The official code for CoT / ZSL reasoning framework 🧠, utilized in paper: \"Large Language Models in Targeted Sentiment Analysis in Russian\"","archived":false,"fork":false,"pushed_at":"2024-09-08T20:18:30.000Z","size":189,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-09-08T21:35:29.239Z","etag":null,"topics":["chainofthought","cot","engine","framework","generative-model","gpt","gpt-35","gpt-4","large-language-models","llm","openai-api","sentiment-analysis","target-sentiment-classification","transformers-models"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2404.12342","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicolay-r.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGESET.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-23T10:12:05.000Z","updated_at":"2024-09-08T20:18:33.000Z","dependencies_parsed_at":"2024-04-08T21:48:29.377Z","dependency_job_id":"f0ae75fa-5707-4c66-99d4-5a9da1a7350a","html_url":"https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework","commit_stats":null,"previous_names":["nicolay-r/reasoning-for-sentiment-analysis-framework"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2FReasoning-for-Sentiment-Analysis-Framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2FReasoning-for-Sentiment-Analysis-Framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2FReasoning-for-Sentiment-Analysis-Framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicolay-r%2FReasoning-for-Sentiment-Analysis-Framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicolay-r","download_url":"https://codeload.github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230374271,"owners_count":18216044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chainofthought","cot","engine","framework","generative-model","gpt","gpt-35","gpt-4","large-language-models","llm","openai-api","sentiment-analysis","target-sentiment-classification","transformers-models"],"created_at":"2024-12-19T04:13:24.572Z","updated_at":"2025-08-19T23:31:16.166Z","avatar_url":"https://github.com/nicolay-r.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reasoning for Sentiment Analysis • [![twitter](https://img.shields.io/twitter/url/https/shields.io.svg?style=social)](https://twitter.com/nicolayr_/status/1781330684289658933)\n![](https://img.shields.io/badge/Python-3.10-brightgreen.svg)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb)\n[![arXiv](https://img.shields.io/badge/arXiv-2404.12342-b31b1b.svg)](https://arxiv.org/abs/2404.12342)\n\nThis repository represent studies and Collection of **LLM-powered reasoning frameworks** 🧠 for **Target Sentiment Analysis**.\nIt contains source code for paper @ [LJoM journal](https://link.springer.com/journal/12202) titled as:\n[Large Language Models in Targeted Sentiment Analysis for Russian](https://arxiv.org/abs/2404.12342).\n\n## Videos\n\n\u003cdiv align=\"center\"\u003e\n\n[![](https://markdown-videos-api.jorgenkh.no/youtube/kdkZsTQibm0)](https://youtu.be/kdkZsTQibm0)\n[![](https://markdown-videos-api.jorgenkh.no/youtube/qawLJsRHzB4)](https://youtu.be/qawLJsRHzB4)\n\n\u003c/div\u003e \n\n\n\u003cdetails\u003e\n\u003csummary\u003e\n\n### **Updates**   \n\n\u003e **Update February 23 2025:** 🔥 Batching mode support. See 🌌 [Flan-T5 provider](https://github.com/nicolay-r/nlp-thirdgate/blob/master/llm/transformers_flan_t5.py) for [bulk-chain](https://github.com/nicolay-r/bulk-chain) project. Test [is available here](https://github.com/nicolay-r/bulk-chain/blob/master/test/test_provider_batching.py)\n\n\u003e **Update November 01 2024:** ⭐ Implemented a separated [bulk-chain](https://github.com/nicolay-r/bulk-chain) project for handling massive amount of prompts with CoT. This concept was used in this studies.\n\n...\n\n\u003c/summary\u003e\n\n\u003e **Update 06 September 2024:** Mentioning the related information about the project at [BU-research-blog](https://blogs.bournemouth.ac.uk/research/2024/09/06/presenting-studies-on-llms-reasoning-capabilities-in-sentiment-analysis-of-mass-media-texts-at-nlpsummit-2024/)\n\n\u003e **Update 11 August 2024:** 🎤 Announcing the talk on this framework @ [NLPSummit 2024](https://www.nlpsummit.org/nlp-summit-2024/) with the preliminary ad and details in [X/Twitter post](https://x.com/JohnSnowLabs/status/1821999936478572836) 🐦. [![twitter](https://img.shields.io/twitter/url/https/shields.io.svg)](https://x.com/JohnSnowLabs/status/1821999936478572836)\n\n\u003e **Update 05 June 2024:** The frameworkless lauch of the **THoR-tuned FlanT5 model** for direct inference is now available in [GoogleColab](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/FlanT5_Finetuned_Model_Usage.ipynb).\n\n\u003e **Update 02 June 2024:** 🤗 Models were uploaded on `huggingface`: [🤗 nicolay-r/flan-t5-tsa-prompt-xl](https://huggingface.co/nicolay-r/flan-t5-tsa-thor-xl) and other models smaller sizes as well. Check out the [related section](#fine-tuned-flan-t5).\n\n\u003e **Update 22 May 2024:** ⚠️ in `GoogleColab` you might find zero evaluation results (see [#8 issue](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/issues/8)) (`0`) on `test` part, which is due to locked availablity on labels 🔐. In order to apply your model for `test` part, please proceed with the [Official RuSentNE-2023 codalab competition page](https://codalab.lisn.upsaclay.fr/competitions/9538) or at [Github](https://github.com/dialogue-evaluation/RuSentNE-evaluation).\n\n\u003e **Update 06 March 2024**: 🔓 `attrdict` represents the main limitation for code launching in `Python 3.10` and hence been switched to `addict` (see [Issue#7](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/issues/7)).\n\n\u003e **🔥 Update 24/04/2024:** We released fine-tuning log for the [prompt-based](DOC_FLANT5_PROMPT_FT.md) and [THoR-based](DOC_FLANT5_THOR_FT.md) techniques applied for the `train` competition data as well as **[checkpoints](#fine-tuned-flan-t5)** for downloading. [More ...](#fine-tuned-flan-t5)\n\n\u003e **💻 Update 19/04/2024:** We open [quick_cot](https://github.com/nicolay-r/quick_cot) code repository for lauching quick CoT zero-shot-learning / few-shot-learning experiments with LLM, utilied in this studies. [More ...](https://github.com/nicolay-r/quick_cot)\n\n\u003e **📊 Update 19/04/2024:** We open a separate 📊 👉[RuSentNE-benchmark repository](https://github.com/nicolay-r/RuSentNE-LLM-Benchmark)👈 📊 for LLM-resonses, including **answers on reasoning steps in THoR CoT** for ChatGPT model series.\n\u003e [More ...](https://github.com/nicolay-r/RuSentNE-LLM-Benchmark)\n\n\u003c/details\u003e\n\n\n## Contents\n\n* [Installation](#installation)\n* [Preparing Data](#preparing-data)\n    * [Manual Translation](#manual-data-translation)\n* [**Zero-Shot**](#zero-shot)\n    * [Examples](#usage-examples)\n* [**Chain-of-Thought fine-tuning**](#three-hop-chain-of-thought-thor)\n* [**Fine-tuned Flan-T5 Checkpoints 🔥**](#fine-tuned-flan-t5)\n* [**Answers**](#answers)\n* [References](#references)\n\n## Installation\n\nWe separate dependencies necessary for zero-shot and fine-tuning experiments:\n```bash\npip install -r dependencies_zs.txt\npip install -r dependencies_ft.txt\n```\n\n## Preparing Data \n\nSimply launch the following script for obtaining both original texts and **Translated**:\n```bash\npython rusentne23_download.py\n```\n\n## Manual Data Translation\nYou could launch manual data translation to English language (`en`) via [GoogleTrans](https://github.com/ssut/py-googletrans):\n```bash\n python rusentne23_translate.py --src \"data/train_data.csv\" --lang \"en\" --label\n python rusentne23_translate.py --src \"data/valid_data.csv\" --lang \"en\" --label\n python rusentne23_translate.py --src \"data/final_data.csv\" --lang \"en\"\n ```\n\n# Zero-Shot\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb)\n[![](https://img.shields.io/badge/prompts-ffffff.svg)](utils_prompt.py)\n\n\nThis is a common script for launching LLM model inference in Zero-shot format using manual or \n[predefined prompts](utils_prompt.py):\n\n```bash\npython zero_shot_infer.py \\\n    --model \"google/flan-t5-base\" \\\n    --src \"data/final_data_en.csv\" \\\n    --prompt \"rusentne2023_default_en\" \\\n    --device \"cpu\" \\\n    --to \"csv\" \\\n    --temp 0.1 \\\n    --output \"data/output.csv\" \\\n    --max-length 512 \\\n    --hf-token \"\u003cYOUR_HUGGINGFACE_TOKEN\u003e\" \\\n    --openai-token \"\u003cYOUR_OPENAI_TOKEN\u003e\" \\\n    --limit 10000 \\\n    --limit-prompt 10000 \\\n    --bf16 \\\n    --l4b\n```\n\n### Notes\n\n* **Mixtral-7B**: [you need to install `transformers 4.36.0`, read the blog post](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/discussions/9); \n  [[model size]](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/discussions/3#657721f02eb103d91fd044f1)\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\n\n### Usage Examples\n\n\u003c/summary\u003e\n\n#### Chat mode\n\nSimply setup `model` name and `device` you wish to use for launching model.\n\n```bash\npython zero_shot_infer.py --model google/flan-t5-base --device cpu\n```\n\n#### Inference with the predefined prompt \n\nUse the `prompt` command for passing the [predefined prompt](utils_prompt.py) or textual prompt that involves the `{text}` information. \n\n```bash\npython zero_shot_infer.py --model google/flan-t5-small \\\n    --device cpu --src data/final_data_en.csv --prompt 'rusentrel2023_default_en'\n```\n\n#### OpenAI models\n\nUse the `model` parameter prefixed by `openai:`, followed by \n[model names](https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/b8e588e4722c27c88acd33bbaeabeee00a903688/zero_shot_infer.py#L63-L79) \nas follows:\n\n```bash\npython zero_shot_infer.py --model \"openai:gpt-3.5-turbo-1106\" \\\n    --src \"data/final_data_en.csv\" --prompt \"rusentrel2023_default_en_short\" \\\n    --max-length 75 --limit 5\n```\n\n\u003c/details\u003e\n\n# Zero-Shot Chain-of-Thought \n\nThis functionality if out-of-scope of this repository.\n\nWe release a tiny framework, dubbed as [quick_cot](https://github.com/nicolay-r/quick_cot) for applying CoT schemas, with API similar to one in  [**Zero-Shot**](#zero-shot) section, based on **schemas** written in JSON notation.\n\n### 📝 👉 [`thor-zero-shot-cot-english-shema.json`](config/thor-zero-shot-cot-english-schema.json) 👈\n### 💻 👉 [Tiny CoT-framework (quick_cot)](https://github.com/nicolay-r/quick_cot) 👈\n\n# Fine-tuned Flan-T5\n\n👉 [**Prompt-Fine-Tuning** Logs](DOC_FLANT5_PROMPT_FT.md)\n\n👉 [**THoR-Fine-Tuning** Logs](DOC_FLANT5_THOR_FT.md)\n\n\n|Model|prompt|THoR|\n|-----|------|----|\n|**FlanT5-base** | - |[🤗 nicolay-r/flan-t5-tsa-thor-base](https://huggingface.co/nicolay-r/flan-t5-tsa-thor-base)|\n|**FlanT5-large**| - |[🤗 nicolay-r/flan-t5-tsa-thor-large](https://huggingface.co/nicolay-r/flan-t5-tsa-thor-large) |\n|**FlanT5-xl**   | [🤗 nicolay-r/flan-t5-tsa-prompt-xl](https://huggingface.co/nicolay-r/flan-t5-tsa-prompt-xl) | [🤗 nicolay-r/flan-t5-tsa-prompt-xl](https://huggingface.co/nicolay-r/flan-t5-tsa-thor-xl)|\n\n\n# Three Hop Chain-of-Thought THoR  \n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb)\n\n```bash\npython thor_finetune.py -r \"thor\" -d \"rusentne2023\" \n    -m \"google/flan-t5-base\" \\\n    -li \u003cPRETRAINED_STATE_INDEX\u003e \\\n    -bs \u003cBATCH_SIZE\u003e \\\n    -es \u003cEPOCH_SIZE\u003e \\\n    -f \"./config/config.yaml\" \n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\n\n### Parameters list\n\u003c/summary\u003e\n\n* `-c`, `--cuda_index`: Index of the GPU to use for computation (default: `0`).\n* `-m`, `--model_path`: Path to the model on hugging face.\n* `-d`, `--data_name`: Name of the dataset (`rusentne2023`)\n* `-r`, `--reasoning`: Specifies the reasoning mode (engine), with single `prompt` or multi-step `thor` mode.\n* `-li`, `--load_iter`: load a state on specific index from the same `data_name` resource (default: `-1`, not applicable.)\n* `-es`, `--epoch_size`: amount of training epochs (default: `1`)\n* `-bs`, `--batch_size`: size of the batch (default: `None`)\n* `-t`, `--temperature`: temperature (default=gen_config.temperature)\n* `-z`, `--zero_shot`: running zero-shot inference with chosen engine on `test` dataset to form answers.\n* `-f`, `--config`: Specifies the location of [config.yaml](config/config.yaml) file.\n\nConfigure more parameters in [config.yaml](config/config.yaml) file.\n\n\u003c/details\u003e\n\n## Answers\n\nResults of the zero-shot models obtained during experiments fall outside the scope of this repository.\nWe open a separate  for LLM-resonses, including **answers on reasoning steps in THoR CoT** for ChatGPT model series:\n\n### 👉 [RuSentNE-benchmark repository](https://github.com/nicolay-r/RuSentNE-LLM-Benchmark) 👈\n\n## References\n\nYou can cite this work as follows:\n```bibtex\n@misc{rusnachenko2024large,\n      title={Large Language Models in Targeted Sentiment Analysis}, \n      author={Nicolay Rusnachenko and Anton Golubev and Natalia Loukachevitch},\n      year={2024},\n      eprint={2404.12342},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicolay-r%2Freasoning-for-sentiment-analysis-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicolay-r%2Freasoning-for-sentiment-analysis-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicolay-r%2Freasoning-for-sentiment-analysis-framework/lists"}