{"id":22069131,"url":"https://github.com/jamesjg/FoodSAM","last_synced_at":"2025-07-24T07:32:27.930Z","repository":{"id":187484990,"uuid":"676947926","full_name":"jamesjg/FoodSAM","owner":"jamesjg","description":"FoodSAM: Any Food Segmentation","archived":false,"fork":false,"pushed_at":"2023-11-06T05:38:08.000Z","size":14327,"stargazers_count":103,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2023-11-06T06:32:11.668Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jamesjg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-08-10T11:35:36.000Z","updated_at":"2023-11-01T14:42:24.000Z","dependencies_parsed_at":"2023-11-06T06:41:15.502Z","dependency_job_id":null,"html_url":"https://github.com/jamesjg/FoodSAM","commit_stats":null,"previous_names":["jamesjg/foodsam"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesjg%2FFoodSAM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesjg%2FFoodSAM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesjg%2FFoodSAM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesjg%2FFoodSAM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jamesjg","download_url":"https://codeload.github.com/jamesjg/FoodSAM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227421429,"owners_count":17775011,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-30T20:04:41.961Z","updated_at":"2024-11-30T20:07:44.271Z","avatar_url":"https://github.com/jamesjg.png","language":"Python","funding_links":[],"categories":["Paper List"],"sub_categories":["Follow-up Papers"],"readme":"# FoodSAM: Any Food Segmentation\n\n\nThis is the official PyTorch implementation of our paper:\nFoodSAM: Any Food Segmentation.\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/foodsam-any-food-segmentation/semantic-segmentation-on-foodseg103)](https://paperswithcode.com/sota/semantic-segmentation-on-foodseg103?p=foodsam-any-food-segmentation)\n\n---\n\nSegment anything Model(SAM) demonstrates significant performance on various segmentation benchmarks, showcasing its impressing zero-shot transfer capabilities on 23 diverse segmentation datasets. However, SAM lacks the class-specific information for each mask. To address the above limitation and explore the zero-shot capability of the SAM for food image segmentation, we propose a novel framework, called FoodSAM. This innovative approach integrates the coarse semantic mask with SAM-generated masks to enhance semantic\nsegmentation quality. Besides, it can perform instance segmentation on food images. Furthermore, FoodSAM extends its zero-shot capability to encompass panoptic segmentation by incorporating an object detector, which renders FoodSAM to effectively capture non-food object information. Remarkably, this pioneering framework stands as the first-ever work to achieve instance, panoptic, and promptable segmentation on food images. \n\n[[`Arxiv`](https://arxiv.org/abs/2308.05938)] \n[[`Project`]](https://starhiking.github.io/FoodSAM_Page/)\n[[`IEEE TMM`]](https://ieeexplore.ieee.org/document/10306316)\n\n![FoodSAM architecture](assets/foodsam.jpg)\n\nFoodSAM contains three basic models: SAM, semantic segmenter, and object detector. SAM generates many class-agnostic binary masks, the semantic segmenter provides food category labels via mask-category match, and the object detector provides the non-food class for background masks. It then enhances the semantic mask via merge strategy and produces instance and panoptic results. Moreover, a seamless prompt-prior selection is integrated into the object detector to achieve promptable segmentation.\n\n  \u003cimg src=\"assets/model.jpg\" /\u003e\n\n## Installation\nPlease follow our [installation.md](installation.md) to install.\n\n\n## \u003ca name=\"GettingStarted\"\u003e\u003c/a\u003eGetting Started\n\n### Demo shell\nYou can run the model for semantic and panoptic segmentation in a few command lines.\n\n#### semantic segmentation:\n\n    # semantic segmentation for one img\n    python FoodSAM/semantic.py --img_path \u003cpath/to/img\u003e --output \u003cpath/to/output\u003e \n\n    # semantic segmentation for one folder\n    python FoodSAM/semantic.py --data_root \u003cpath/to/folder\u003e --output \u003cpath/to/output\u003e\n\n#### panoptic segmentation:\n\n    # panoptic segmentation for one img\n    python FoodSAM/panoptic.py --img_path \u003cpath/to/img\u003e --output \u003cpath/to/output\u003e\n\n    # panoptic segmentation for one folder\n    python FoodSAM/panoptic.py --data_root \u003cpath/to/folder\u003e --output \u003cpath/to/output\u003e\n\n\n\n### Evaluation shell\nFurthermore, by setting `args.eval` to true, the model can output the semantic masks and evaluate the metrics. \nHere are examples of semantic segmentation and panoptic segmentation on the FoodSeg103 dataset:\n```\npython FoodSAM/semantic.py --data_root dataset/FoodSeg103/Images --output Output/Semantic_Results --eval \n```\n```\npython FoodSAM/panoptic.py --data_root dataset/FoodSeg103/Images --output Output/Panoptic_Results\n```\n\n## Quantitative results\n\n### FoodSeg103\n| Method | mIou | aAcc | mAcc \n| :-: | :- | -: | :-: |  \n|[SETR_MLA(baseline)](https://github.com/LARC-CMU-SMU/FoodSeg103-Benchmark-v1) | 45.10 | 83.53 | 57.44\nFoodSAM | 46.42 | 84.10 |  58.27\n\n### UECFOODPIXCOMPLETE\n\n| Method | mIou | aAcc | mAcc \n| :-: | :- | -: | :-: |  \n|[deeplabV3+ (baseline)](https://github.com/HitBadTrap/Foodseg-uecfoodpix)| 65.61 |88.20| 77.56\nFoodSAM | 66.14 |88.47 |78.01\n\n## Qualitative results\n\n### cross domain results\n\n \u003cimg src=\"assets/crossdomain.png\"\u003e\n\n### semantic segmentation results \n\n \u003cimg src=\"assets/semantic.jpg\"\u003e\n \n---\n\n \u003cimg src=\"assets/semantic_compare.jpg\"\u003e\n \n### instance segmentation results\n\u003cimg src=\"assets/instance_compare.jpg\"\u003e\n\n### panoptic segmentation results\n\u003cimg src=\"assets/panoptic_compare.jpg\"\u003e\n\n### promptable segmentation results\n\u003cimg src=\"assets/prompt_vis.jpg\"\u003e\n\n## Acknowledgements\n\nA large part of the code is borrowed from the following wonderful works:\n\n1. [Segmentation Anything](https://github.com/facebookresearch/segment-anything)\n\n2. [UniDet](https://github.com/xingyizhou/UniDet)\n\n3. [FoodSeg103](https://github.com/LARC-CMU-SMU/FoodSeg103-Benchmark-v1)\n\n4. [mmsegmentation](https://github.com/open-mmlab/mmsegmentation)\n\n## License\n\nThe model is licensed under the [Apache 2.0 license](LICENSE).\n\n## Citation\nIf you want to cite our work, please use this:\n\n```\n@ARTICLE{10306316,\n  author={Lan, Xing and Lyu, Jiayi and Jiang, Hanyu and Dong, Kun and Niu, Zehai and Zhang, Yi and Xue, Jian},\n  journal={IEEE Transactions on Multimedia}, \n  title={FoodSAM: Any Food Segmentation}, \n  year={2023},\n  volume={},\n  number={},\n  pages={1-14},\n  doi={10.1109/TMM.2023.3330047}\n}\n\n@misc{lan2023foodsam,\n      title={FoodSAM: Any Food Segmentation}, \n      author={Xing Lan and Jiayi Lyu and Hanyu Jiang and Kun Dong and Zehai Niu and Yi Zhang and Jian Xue},\n      year={2023},\n      eprint={2308.05938},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesjg%2FFoodSAM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjamesjg%2FFoodSAM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesjg%2FFoodSAM/lists"}