{"id":23726865,"url":"https://github.com/depshad/deep-learning-framework-for-multi-modal-product-classification","last_synced_at":"2025-09-04T03:32:25.471Z","repository":{"id":197712210,"uuid":"283237233","full_name":"depshad/Deep-Learning-Framework-for-Multi-modal-Product-Classification","owner":"depshad","description":"Code repository for Rakuten Data Challenge: Multimodal Product Classification and Retrieval. ","archived":false,"fork":false,"pushed_at":"2021-05-10T17:40:41.000Z","size":178,"stargazers_count":23,"open_issues_count":0,"forks_count":9,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-10-03T13:07:32.498Z","etag":null,"topics":["computer-vision","deep-learning","multi-modal-learning","nlp","pytorch","rakuten-data-challenge"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/depshad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-07-28T14:30:29.000Z","updated_at":"2023-09-13T14:09:50.000Z","dependencies_parsed_at":"2023-10-02T06:17:29.795Z","dependency_job_id":null,"html_url":"https://github.com/depshad/Deep-Learning-Framework-for-Multi-modal-Product-Classification","commit_stats":null,"previous_names":["depshad/deep-learning-framework-for-multi-modal-product-classification"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/depshad%2FDeep-Learning-Framework-for-Multi-modal-Product-Classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/depshad%2FDeep-Learning-Framework-for-Multi-modal-Product-Classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/depshad%2FDeep-Learning-Framework-for-Multi-modal-Product-Classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/depshad%2FDeep-Learning-Framework-for-Multi-modal-Product-Classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/depshad","download_url":"https://codeload.github.com/depshad/Deep-Learning-Framework-for-Multi-modal-Product-Classification/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231929831,"owners_count":18447581,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","multi-modal-learning","nlp","pytorch","rakuten-data-challenge"],"created_at":"2024-12-31T00:36:08.110Z","updated_at":"2024-12-31T00:36:08.769Z","avatar_url":"https://github.com/depshad.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deep Learning Framework for Multi-modal Product Classification\nCode repository for Rakuten Data Challenge : Multimodal Product Classification and Retrieval. \n\nTeam Transformer's solution : Deep Multi-level Boosted Fusion Learning Framework for Multi-modal Product Classification \n \nPaper Link : https://sigir-ecom.github.io/ecom20DCPapers/SIGIR_eCom20_DC_paper_8.pdf\n\n\nData challenge link : https://sigir-ecom.github.io/data-task.html\n\n## Abstract\n\nIn this paper, we present our approach for the ’Multimodal Product \nClassification’ task as a part of the 2020 SIGIR Workshop On eCommerce (ECOM20). The specific objective of this task is to build and\nsubmit systems that classify previously unseen products into their\ncorresponding product type codes. We propose a deep Multi-Modal\nMulti-level Boosted Fusion Learning Framework used to categorize\nlarge-scale multi-modal (text and image) product data into product\ntype codes. Our proposed final methodology achieved a macro F1-\nscore of 91.94 on the phase 1 test dataset which is the top-scoring\nsubmission and third position on the scoreboard for phase 2 test\ndataset with macro F1-score of 90.53.\n\n## Code Usage\n\n### Unimodal Model Training and Prediction Scripts\n\n1. SEResnext50_train_predict.ipynb : Fine tune the pre-trained SEResnext50 model on Rakuten images\n\n2. camembert_train_predict.ipynb : Fine tune the pre-trained Cammebert model on French text; Custom Cammbert model with vector output (used later for feature fusion)\n\n3. flaubert_train_predict.ipynb : Fine tune the pre-trained Flaubert model on French text; Custom Flaubert model with vector output (used later for feature fusion)\n\n### Multimodal Feature Level Fusion\n1. multi-modal_concatenate_fusion.ipynb : Concatenate the features extracted and train NN module on top\n\n### Probability Level Fusion\n1. Boosted Late-Fusion.ipynb : Train LightGBM model with class probability as input\n\n\n\n\u003cp align=\"center\"\u003e Multi-modal Joint Representation Learning \u003c/p\u003e \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/56831322/89715638-a5ff2280-d9c4-11ea-9ca1-be884c8b9c26.png\" /\u003e\n\u003c/p\u003e\n\n\n\n\u003cp align=\"center\"\u003e Late Fusion Model \u003c/p\u003e \n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/56831322/89715668-f1193580-d9c4-11ea-8fcd-042e909ee30d.png\" /\u003e\n\u003c/p\u003e\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdepshad%2Fdeep-learning-framework-for-multi-modal-product-classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdepshad%2Fdeep-learning-framework-for-multi-modal-product-classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdepshad%2Fdeep-learning-framework-for-multi-modal-product-classification/lists"}