{"id":30104252,"url":"https://github.com/ggcr/tinyemo","last_synced_at":"2025-08-09T22:33:33.655Z","repository":{"id":277112289,"uuid":"866092708","full_name":"ggcr/TinyEmo","owner":"ggcr","description":"\"TinyEmo: Scaling down Emotional Reasoning via Metric Projection\" ACMCV 2024","archived":false,"fork":false,"pushed_at":"2025-02-12T06:57:38.000Z","size":4413,"stargazers_count":8,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-12T07:53:22.731Z","etag":null,"topics":["affective-computing","visual-emotion-recognition","visual-sentiment-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ggcr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-01T16:26:58.000Z","updated_at":"2025-02-12T06:57:42.000Z","dependencies_parsed_at":"2025-02-12T08:03:40.794Z","dependency_job_id":null,"html_url":"https://github.com/ggcr/TinyEmo","commit_stats":null,"previous_names":["ggcr/tinyemo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ggcr/TinyEmo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggcr%2FTinyEmo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggcr%2FTinyEmo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggcr%2FTinyEmo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggcr%2FTinyEmo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ggcr","download_url":"https://codeload.github.com/ggcr/TinyEmo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ggcr%2FTinyEmo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269649304,"owners_count":24453499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-09T02:00:10.424Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["affective-computing","visual-emotion-recognition","visual-sentiment-analysis"],"created_at":"2025-08-09T22:33:32.801Z","updated_at":"2025-08-09T22:33:33.636Z","avatar_url":"https://github.com/ggcr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TinyEmo\n\n[[Paper]](https://arxiv.org/abs/2410.07062)\n\n[[Metric Projector Card]](https://huggingface.co/collections/ggcristian/tinyemo-projectors-66fd14187fbd5d30764abc24) [TinyEmo MM-LLM Card]\n\n[[Dataset card]](https://huggingface.co/collections/ggcristian/tinyemo-emoreason-dataset-66fd16963c945fb7058a8f55)\n\nTinyEmo is a family of small multi-modal language models for emotional reasoning and classification. Our\napproach features: (1) a synthetic emotional instruct dataset for both pre-training and fine-tuning stages, (2) a Metric Projector\nthat delegates classification from the language model allowing for more efficient training and inference, (3) a multi-modal large\nlanguage model (MM-LLM) for emotional reasoning, and (4) a semi-automated framework for bias detection. TinyEmo is able to\nperform emotion classification and emotional reasoning, all while using substantially fewer parameters than comparable models.\nThis efficiency allows us to freely incorporate more diverse emotional datasets, enabling strong performance on classification tasks,\nwith our smallest model (700M parameters) outperforming larger state-of-the-art models based on general-purpose MM-LLMs\nwith over 7B parameters. Additionally, the Metric Projector allows for interpretability and indirect bias detection in large models\nwithout additional training, offering an approach to understand and improve AI systems.\n\n## Installation and Requirements\n\n### Metric Projector (Classification)\n\n1. Clone this repository and navigate to the root of the project:\n```\ngit clone https://github.com/ggcr/TinyEmo.git\ncd TinyEmo\n```\n\n2. Create an environment and install dependencies:\n```\nconda create -n projector_mps python=3.10 -y\nconda activate projector_mps\npip install --upgrade pip  # enable PEP 660 support\npip install -e projector_mps/.\n```\n\n### MM-LLM (Reasoning)\n\nRefer to the [TinyLLaVA](https://github.com/TinyLLaVA/TinyLLaVA_Factory) installation section.\n\n\n\n## Quickstart\n\n### Metric Projector inference\n\nWe provide precomputed CLIP features for the Emotion6 dataset, and you can evaluate them using two methods:\n\n#### Our Projectors from Hugging Face\n\nTo evaluate the projectors from Hugging Face, use the [scripts/eval.sh](https://github.com/ggcr/TinyEmo/blob/main/projector_mps/scripts/eval.sh) script:\n\n```bash\nconda activate projector_mps\nbash projector_mps/scripts/eval.sh\n```\n\nThe **Zero-shot Accuracy** in the table below is the average accuracy across multiple datasets, including *Emotion6*, *FI*, *ArtPhoto*, *Abstract*, and *UnbiasedEmo*.\n\n| Model Architecture                     | Parameters | Zero-shot Accuracy | HuggingFace Link                                                                 |\n|----------------------------------------| ---------- |--------------------|----------------------------------------------------------------------|\n| CLIP ViT-L/14 + OpenELM-270M-I         | 0.70B      | 57.87%             | [HF Projector 0.70B Link](https://huggingface.co/ggcristian/TinyEmo-CLIP-OpenELM-270M) |\n| CLIP ViT-L/14 + OpenELM-450M-I         | 0.88B      | 55.24%             | [HF Projector 0.88B Link](https://huggingface.co/ggcristian/TinyEmo-CLIP-OpenELM-450M) |\n| CLIP ViT-L/14 + TinyLLaMA 1.1          | 1.53B      | 56.13%             | [HF Projector 1.53B Link](https://huggingface.co/ggcristian/TinyEmo-CLIP-TinyLlama-1_1-Syn) |\n| CLIP ViT-L/14 + Microsoft Phi 2        | 3.21B      | 56.28%             | [HF Projector 3.21B Link](https://huggingface.co/ggcristian/TinyEmo-CLIP-Phi-2)      |\n\nA more extensive eval of the results can be seen on Table VIII from the paper:\n\n\u003cimg src=\"projector_mps/misc/images/table_viii.png\" /\u003e\n\n#### Custom Projectors with Local Weights\n\nTo use custom local weights or models, run the following:\n\n```bash\nconda activate projector_mps\nbash projector_mps/scripts/eval_custom.sh\n```\n\nThis allows you to specify different vision encoders, language models, and loss functions, as well as use your own projector weights.\n\n\n## Acknowledgement\n\nThe Metric Projector was built from the foundations of [CLIP-E](https://arxiv.org/abs/2310.12062) paper!\n\nOur codebase for the MM-LLM is forked from the [TinyLLaVA](https://github.com/TinyLLaVA/TinyLLaVA_Factory) project.\n\n## Citation\n\n```\n@mastersthesis{gutierrez2024tinyemo,\n  title        = {TinyEmo: Scaling down Emotional Reasoning via Metric Projection},\n  author       = {Cristian Gutierrez},\n  year         = 2024,\n  month        = {September},\n  address      = {Barcelona, Spain},\n  note         = {Available at \\url{https://arxiv.org/abs/2410.07062}},\n  school       = {Universitat Autonoma de Barcelona (UAB)},\n  type         = {Master's thesis in Computer Vision}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fggcr%2Ftinyemo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fggcr%2Ftinyemo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fggcr%2Ftinyemo/lists"}