{"id":26849145,"url":"https://github.com/hiborn4/tensorfusion_network_for_multimodal_sentiment_analysis","last_synced_at":"2025-06-29T00:33:40.334Z","repository":{"id":240891476,"uuid":"803701379","full_name":"HiBorn4/TensorFusion_Network_for_Multimodal_sentiment_analysis","owner":"HiBorn4","description":"This repository implements the Tensor Fusion Network (TFN) for multimodal sentiment analysis using the CMU-MOSI dataset. TFN integrates language, visual, and acoustic modalities to predict sentiment intensity, enhancing sentiment prediction accuracy by modeling unimodal, bimodal, and trimodal interactions.","archived":false,"fork":false,"pushed_at":"2024-05-21T08:26:17.000Z","size":398,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-30T21:33:20.535Z","etag":null,"topics":["cmu-mosi","deep","deep-learning","fusion","lrmf","regression","tensorflow","unimodals"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HiBorn4.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-21T08:19:44.000Z","updated_at":"2025-01-09T13:53:12.000Z","dependencies_parsed_at":"2024-05-21T10:10:32.752Z","dependency_job_id":"d6d9c440-dbd3-4b02-b39f-a16791555c80","html_url":"https://github.com/HiBorn4/TensorFusion_Network_for_Multimodal_sentiment_analysis","commit_stats":null,"previous_names":["hiborn4/tensorfusion_network_for_multimodal_sentiment_analysis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HiBorn4/TensorFusion_Network_for_Multimodal_sentiment_analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HiBorn4%2FTensorFusion_Network_for_Multimodal_sentiment_analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HiBorn4%2FTensorFusion_Network_for_Multimodal_sentiment_analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HiBorn4%2FTensorFusion_Network_for_Multimodal_sentiment_analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HiBorn4%2FTensorFusion_Network_for_Multimodal_sentiment_analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HiBorn4","download_url":"https://codeload.github.com/HiBorn4/TensorFusion_Network_for_Multimodal_sentiment_analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HiBorn4%2FTensorFusion_Network_for_Multimodal_sentiment_analysis/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262518102,"owners_count":23323301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cmu-mosi","deep","deep-learning","fusion","lrmf","regression","tensorflow","unimodals"],"created_at":"2025-03-30T21:32:28.074Z","updated_at":"2025-06-29T00:33:40.307Z","avatar_url":"https://github.com/HiBorn4.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tensor Fusion Network (TFN) for Multimodal Sentiment Analysis\n\nThis repository contains the implementation of the Tensor Fusion Network (TFN) for multimodal sentiment analysis using the CMU-MOSI dataset. The TFN architecture incorporates language, visual, and acoustic modalities to predict sentiment intensity.\n\n## Dataset: CMU-MOSI\n\nThe CMU-MOSI dataset is an annotated dataset of video opinions from YouTube movie reviews. It includes sentiment annotations on a seven-step Likert scale from very negative to very positive. The dataset comprises 2,199 opinion utterances from 93 distinct speakers, with an average length of 4.2 seconds per video.\n\n### Dataset Features\n\n- **Language Modality**: Uses GloVe word vectors for spoken words.\n- **Visual Modality**: Extracts facial expressions and action units using the FACET framework and OpenFace.\n- **Acoustic Modality**: Extracts acoustic features using the COVAREP framework.\n\n### Sentiment Prediction Tasks\n\n1. **Binary Sentiment Classification**\n2. **Five-Class Sentiment Classification**\n3. **Sentiment Regression**\n\n## Tensor Fusion Network (TFN)\n\nTFN consists of three main components:\n\n1. **Modality Embedding Subnetworks**: Extracts features from language, visual, and acoustic modalities.\n2. **Tensor Fusion Layer**: Explicitly models unimodal, bimodal, and trimodal interactions.\n3. **Sentiment Inference Subnetwork**: Performs sentiment inference based on the fused multimodal tensor.\n\n### Modality Embedding Subnetworks\n\n- **Language Embedding Subnetwork**: Uses LSTM to learn time-dependent representations of spoken words.\n- **Visual Embedding Subnetwork**: Uses a deep neural network to process visual features extracted from facial expressions.\n- **Acoustic Embedding Subnetwork**: Uses a deep neural network to process acoustic features extracted from audio signals.\n\n### Tensor Fusion Layer\n\nThe Tensor Fusion Layer models the interactions between different modalities using a three-fold Cartesian product, generating a multimodal tensor that captures unimodal, bimodal, and trimodal dynamics.\n\n### Sentiment Inference Subnetwork\n\nA fully connected deep neural network that takes the multimodal tensor as input and performs sentiment classification or regression.\n\n## Experiments\n\nThree sets of experiments were conducted:\n\n1. **Multimodal Sentiment Analysis**: Compared TFN with state-of-the-art multimodal sentiment analysis models.\n2. **Tensor Fusion Evaluation**: Analyzed the importance of subtensors and the impact of each modality.\n3. **Modality Embedding Subnetworks Evaluation**: Compared TFN's modality-specific networks with state-of-the-art unimodal sentiment analysis models.\n\n## Results\n\nTFN outperformed state-of-the-art approaches in binary sentiment classification, five-class sentiment classification, and sentiment regression. The ablation study showed the importance of modeling trimodal dynamics for improved performance.\n\n## How to Use\n\n### Prerequisites\n\n- Python 3.x\n- TensorFlow or PyTorch (depending on the implementation)\n- Required Python libraries (listed in `requirements.txt`)\n\n### Installation\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/yourusername/TFN-multimodal-sentiment.git\n   cd TFN-multimodal-sentiment\n   ```\n\n2. Install the required Python libraries:\n   ```bash\n   pip3 install -r requirements.txt\n   ```\n\n### Dataset Preparation\n\n1. Download the CMU-MOSI dataset from the official source.\n2. Extract the dataset and place it in the `data` directory.\n\n### Training the Model\n\n1. Preprocess the dataset:\n   ```bash\n   python3 preprocess.py --data_dir data/CMU-MOSI\n   ```\n\n2. Train the TFN model:\n   ```bash\n   python3 train.py --config configs/tfn_config.json\n   ```\n\n### Evaluation\n\nEvaluate the trained model on the test set:\n```bash\npython3 evaluate.py --model_dir models/tfn --data_dir data/CMU-MOSI\n```\n\n### Configuration\n\nModify the configuration file `configs/tfn_config.json` to change hyperparameters, model settings, and dataset paths.\n\n## Citation\n\nIf you use this code or dataset in your research, please cite the original paper:\n\n```bibtex\n@inproceedings{zadeh2016mosi,\n  title={Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages},\n  author={Zadeh, Amir and Chen, Minghai and Poria, Soujanya and Cambria, Erik and Morency, Louis-Philippe},\n  booktitle={IEEE Intelligent Systems},\n  year={2016}\n}\n```\n\n## License\n\nThis project is licensed under the MIT License.\n\n---\n\nFeel free to open an issue if you have any questions or need further assistance. Happy researching!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhiborn4%2Ftensorfusion_network_for_multimodal_sentiment_analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhiborn4%2Ftensorfusion_network_for_multimodal_sentiment_analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhiborn4%2Ftensorfusion_network_for_multimodal_sentiment_analysis/lists"}