{"id":27242387,"url":"https://github.com/animikhaich/deep-convolutional-background-subtractor","last_synced_at":"2025-04-10T20:19:26.511Z","repository":{"id":45926626,"uuid":"387881076","full_name":"animikhaich/Deep-Convolutional-Background-Subtractor","owner":"animikhaich","description":"End-to-end CNN-based Autoencoder that can segment any objects even if it is out of the classes present in the training set. ","archived":false,"fork":false,"pushed_at":"2023-01-03T23:52:04.000Z","size":9293,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-18T08:14:15.428Z","etag":null,"topics":["background-subtraction","cnn","cnn-keras","coco","convolutional-neural-networks","foreground-segmentation","image-segmentation","python","segmentation","tensorflow","unseen-background-subtractor"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/animikhaich.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-20T18:29:59.000Z","updated_at":"2024-01-03T06:31:58.000Z","dependencies_parsed_at":"2023-02-01T14:15:41.168Z","dependency_job_id":null,"html_url":"https://github.com/animikhaich/Deep-Convolutional-Background-Subtractor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animikhaich%2FDeep-Convolutional-Background-Subtractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animikhaich%2FDeep-Convolutional-Background-Subtractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animikhaich%2FDeep-Convolutional-Background-Subtractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/animikhaich%2FDeep-Convolutional-Background-Subtractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/animikhaich","download_url":"https://codeload.github.com/animikhaich/Deep-Convolutional-Background-Subtractor/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248289606,"owners_count":21078917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-subtraction","cnn","cnn-keras","coco","convolutional-neural-networks","foreground-segmentation","image-segmentation","python","segmentation","tensorflow","unseen-background-subtractor"],"created_at":"2025-04-10T20:19:25.986Z","updated_at":"2025-04-10T20:19:26.495Z","avatar_url":"https://github.com/animikhaich.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Contributors][contributors-shield]][contributors-url]\n[![Forks][forks-shield]][forks-url]\n[![Stargazers][stars-shield]][stars-url]\n[![Issues][issues-shield]][issues-url]\n[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n\n\u003c!-- PROJECT LOGO --\u003e\n\u003cbr /\u003e\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"assets/icon.png\" alt=\"Logo\" width=\"150\" height=\"150\"\u003e\n\n  \u003ch2 align=\"center\"\u003eDeep Convolutional Background Subtractor\u003c/h2\u003e\n\n  \u003cp align=\"center\"\u003e\n    A Lightweight Custom Foreground Segmentation Model Trained on Modified COCO.\n    \u003cbr /\u003e\n    \u003ca href=\"https://nbviewer.org/github/animikhaich/single-object-background-subtractor/blob/main/Test_Images.ipynb\"\u003eResults Notebook\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/animikhaich/single-object-background-subtractor/issues/new\"\u003eReport Bug\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/val_result_17.jpg\" alt=\"Demo GIF\"\u003e\n\u003c/p\u003e\n\u003c!-- TABLE OF CONTENTS --\u003e\n\n## Table of Contents\n\n- [Table of Contents](#table-of-contents)\n- [About The Project](#about-the-project)\n- [Jupyter Notebooks - nbViewer](#jupyter-notebooks---nbviewer)\n- [Dataset Information](#dataset-information)\n- [Dataset Preprocessing](#dataset-preprocessing)\n- [Results](#results)\n  - [Masks for Validation Set from COCO (Contains same classes to that in the training set)](#masks-for-validation-set-from-coco-contains-same-classes-to-that-in-the-training-set)\n  - [Inference Results on a Collected Dataset of Guns from Google (A class similar to this was not present in the COCO dataset)](#inference-results-on-a-collected-dataset-of-guns-from-google-a-class-similar-to-this-was-not-present-in-the-coco-dataset)\n- [How to Run](#how-to-run)\n  - [Hardware Used for the Experiment](#hardware-used-for-the-experiment)\n  - [Dataset Directory Structure (For Training)](#dataset-directory-structure-for-training)\n  - [Built With](#built-with)\n- [Changelog](#changelog)\n- [Contributing](#contributing)\n- [License](#license)\n- [Contact](#contact)\n    - [Animikh Aich](#animikh-aich)\n\n\u003c!-- ABOUT THE PROJECT --\u003e\n\n## About The Project\n\nWhile working on synthetic data generation (Like Cut, Paste and Learn: [ArXiv Paper](https://arxiv.org/abs/1708.01642)), one of the many challenges that I faced was an easy way to extract the foreground object of interest from the background image. Having worked with simple UNet-like ([Paper](https://arxiv.org/abs/1505.04597)) architectures in the past, I wanted to test out a hypothesis that an Autoencoder should be able to learn to differentiate foreground from background objects and if trained on enough variety, it would be able to do the same for unseen objects as well (classes it has not seen during training).\n\nThe goal was to train an efficient end-to-end CNN capable of segmenting out the foreground objects, even for classes that are not present in the training set.\n\nEven though I have used COCO dataset for this experiment, there is a good amount of preprocessing that has been done to convert the dataset to a format that suites the need. Hence, if you want to replicate it, be sure to check that part out.\n\nIn summary: the results are quite impressive as the model clearly displays potential to accurately extract foreground objects, both for seen and unseen classes. Please on to find out more!\n\nThe notebooks do not render properly on GitHub, hence please use the [nbviewer](https://nbviewer.jupyter.org/) links provided below to see the results.\n\n## Jupyter Notebooks - nbViewer\n\n- [Dataset Preparation - Extracting the per-object masks from the COOC dataset](https://nbviewer.jupyter.org/github/animikhaich/single-object-background-subtractor/blob/main/Dataset_Preparation.ipynb)\n- [Model - Model Notebook containing Data Loader and Architecture: Training](https://nbviewer.jupyter.org/github/animikhaich/single-object-background-subtractor/blob/main/Model.ipynb)\n- [Results/Testing - Test Results on Validation Set and a collection of Gun Images from Google](https://nbviewer.jupyter.org/github/animikhaich/single-object-background-subtractor/blob/main/Test_Images.ipynb)\n\n## Dataset Information\n\n- The Model is trained on [COCO 2017 Dataset](https://cocodataset.org/).\n- Dataset Splits Used:\n  - Train: COCO 2017 Train Images + Train Annotations - `instances_train2017.json`\n  - Val: COCO 2017 Val Images + Val Annotations - `instances_val2017.json`\n- Dataset Download: https://cocodataset.org/#download\n- Dataset Format Information: https://cocodataset.org/#format-data\n- API to parse COCO: https://github.com/philferriere/cocoapi\n\n## Dataset Preprocessing\n\n- COCO contains instance segmentation annotations, and that was the primary reason behind selecting this dataset.\n- We do not want to have more than one segmentation mask in a given image since our aim is to build network capable of segmenting out the foreground object.\n- We thus process the dataset to crop out the surrounding region of each instance segmentation and save that image and the corresponding mask for that instance as training inputs and targets respectively.\n- Though this does create quite a few bad samples of data, they have been deliberately been left in the dataset to add some noise elements to the dataset and because this is just an experimental project (It was almost morning and I was sleepy, got lazy).\n\n## Results\n\nThere are two sets of results:\n- **First:** Results from the Validation Set of the COCO Dataset (after preprocessing). This contains data from classes which the model has already seen before (present in the training set). For example, dog, human, car, cup, etc.\n- **Second:** The most interesting results are of that class which the model has never seen before (not present in training set). To test that out, I collected a few variety of images of guns from Google Image, and passed them through the network for inference.\n\nI'll let the results speak for itself.\n\n### Masks for Validation Set from COCO (Contains same classes to that in the training set)\n\nImages (Left to Right): `Input Image`, `Predicted Image`, `Thresholded Mask @ 0.5`, `Ground Truth Mask`\n\n![Validation Set Object 1](assets/val_result_2.jpg)\n![Validation Set Object 2](assets/val_result_3.jpg)\n![Validation Set Object 3](assets/val_result_4.jpg)\n![Validation Set Object 4](assets/val_result_9.jpg)\n![Validation Set Object 5](assets/val_result_11.jpg)\n![Validation Set Object 6](assets/val_result_13.jpg)\n![Validation Set Object 7](assets/val_result_15.jpg)\n![Validation Set Object 8](assets/val_result_16.jpg)\n![Validation Set Object 9](assets/val_result_18.jpg)\n![Validation Set Object 10](assets/val_result_19.jpg)\n![Validation Set Object 11](assets/val_result_21.jpg)\n![Validation Set Object 12](assets/val_result_25.jpg)\n![Validation Set Object 13](assets/val_result_28.jpg)\n![Validation Set Object 14](assets/val_result_29.jpg)\n![Validation Set Object 15](assets/val_result_30.jpg)\n![Validation Set Object 16](assets/val_result_31.jpg)\n![Validation Set Object 17](assets/val_result_32.jpg)\n\n### Inference Results on a Collected Dataset of Guns from Google (A class similar to this was not present in the COCO dataset)\n\nImages (Left to Right): `Input Image`, `Predicted Image`, `Thresholded Mask @ 0.5`, `Masked Background (Segmented Object)`\n\n![Out of Class Object 1](assets/gun_result_1.jpg)\n![Out of Class Object 2](assets/gun_result_2.jpg)\n![Out of Class Object 3](assets/gun_result_3.jpg)\n![Out of Class Object 4](assets/gun_result_4.jpg)\n![Out of Class Object 5](assets/gun_result_5.jpg)\n![Out of Class Object 6](assets/gun_result_6.jpg)\n![Out of Class Object 7](assets/gun_result_7.jpg)\n![Out of Class Object 8](assets/gun_result_8.jpg)\n![Out of Class Object 9](assets/gun_result_10.jpg)\n![Out of Class Object 10](assets/gun_result_11.jpg)\n![Out of Class Object 11](assets/gun_result_13.jpg)\n![Out of Class Object 12](assets/gun_result_14.jpg)\n![Out of Class Object 13](assets/gun_result_15.jpg)\n![Out of Class Object 14](assets/gun_result_17.jpg)\n![Out of Class Object 15](assets/gun_result_18.jpg)\n![Out of Class Object 16](assets/gun_result_20.jpg)\n\n## How to Run\n\nThe experiment should be fairly reproducible. However, a GPU would be recommended for training. For Inference, a CPU System would suffice.\n\n### Hardware Used for the Experiment\n\n- CPU: AMD Ryzen 7 3700X - 8 Cores 16 Threads\n- GPU: Nvidia GeForce RTX 2080 Ti 11 GB\n- RAM: 32 GB DDR4 @ 3200 MHz\n- Storage: 1 TB NVMe SSD (This is not important, even a normal SSD would suffice)\n- OS: Ubuntu 20.10\n\nAlternative Option: [Google Colaboratory - GPU Kernel](https://colab.research.google.com/)\n\n### Dataset Directory Structure (For Training)\n\n- Use the COCO API to extract the masks from the dataset. (Refer: [Dataset Preparation.ipynb Notebook](https://nbviewer.jupyter.org/github/animikhaich/single-object-background-subtractor/blob/main/Dataset_Preparation.ipynb))\n- Save the masks in a directory as `.jpg` images.\n- Example Directory Structure:\n\n```sh\n.\n├── images\n│   ├── train\n│   │   ├── *.jpg\n│   └── val\n│       └── *.jpg\n└── masks\n│   ├── train\n│   │   ├── *.jpg\n│   └── val\n│       └── *.jpg\n```\n\n### Built With\n\nSimple List of Deep Learning Libraries. The main Architecture/Model is developed with Keras, which comes as a part of Tensorflow 2.x\n\n- [Tensorflow 2.5.0](https://www.tensorflow.org/)\n- [OpenCV 4.5.1.48](https://opencv.org/)\n- [Numpy 1.19.5](https://numpy.org/)\n- [Matplotlib 3.3.4](https://matplotlib.org/)\n- [PyCOCOTools 2.0.2](https://github.com/philferriere/cocoapi)\n\n\n## Changelog\n\nSince this is a Proof of Concept Project, I am not maintaining a CHANGELOG.md at the moment. However, the primary goal is to improve the architecture to make the predicted masks more accurate.\n\n\n## Contributing\n\nContributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n## License\n\nDistributed under the Copyleft License: GNU AGPL v3. See [LICENSE](LICENSE) for more information.\n\n## Contact\n\n#### Animikh Aich\n\n- Website: [Animikh Aich - Website](http://www.animikh.me/)\n- LinkedIn: [animikh-aich](https://www.linkedin.com/in/animikh-aich/)\n- Email: [animikhaich@gmail.com](mailto:animikhaich@gmail.com)\n- Twitter: [@AichAnimikh](https://twitter.com/AichAnimikh)\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n\n[contributors-shield]: https://img.shields.io/github/contributors/animikhaich/single-object-background-subtractor.svg?style=flat-square\n[contributors-url]: https://github.com/animikhaich/single-object-background-subtractor/graphs/contributors\n[forks-shield]: https://img.shields.io/github/forks/animikhaich/single-object-background-subtractor.svg?style=flat-square\n[forks-url]: https://github.com/animikhaich/single-object-background-subtractor/network/members\n[stars-shield]: https://img.shields.io/github/stars/animikhaich/single-object-background-subtractor.svg?style=flat-square\n[stars-url]: https://github.com/animikhaich/single-object-background-subtractor/stargazers\n[issues-shield]: https://img.shields.io/github/issues/animikhaich/single-object-background-subtractor.svg?style=flat-square\n[issues-url]: https://github.com/animikhaich/single-object-background-subtractor/issues\n[license-shield]: https://img.shields.io/github/license/animikhaich/single-object-background-subtractor.svg?style=flat-square\n[license-url]: https://github.com/animikhaich/single-object-background-subtractor/blob/master/LICENSE.md\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=flat-square\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://linkedin.com/in/animikh-aich/\n[product-screenshot]: assets/face-blur-demo.gif\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanimikhaich%2Fdeep-convolutional-background-subtractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanimikhaich%2Fdeep-convolutional-background-subtractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanimikhaich%2Fdeep-convolutional-background-subtractor/lists"}