{"id":15650835,"url":"https://github.com/nikolasent/personmask_tflite","last_synced_at":"2025-09-06T19:32:31.106Z","repository":{"id":205011077,"uuid":"275260734","full_name":"NikolasEnt/PersonMask_TFLite","owner":"NikolasEnt","description":"Real-time CPU person segmentation for privacy in video calls","archived":false,"fork":false,"pushed_at":"2020-06-26T23:24:19.000Z","size":5875,"stargazers_count":37,"open_issues_count":0,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-09-01T04:21:40.432Z","etag":null,"topics":["c-plus-plus","deep-learning","mobilenetv3","portrait-segmentation","semantic-segmentation","tensorflow-lite","tflite","v4l2","v4l2loopback","video","video-calls","video-processing"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NikolasEnt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-06-26T22:45:59.000Z","updated_at":"2025-03-16T11:13:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"cd236c49-3d93-439a-9e1c-3d022aca3f55","html_url":"https://github.com/NikolasEnt/PersonMask_TFLite","commit_stats":null,"previous_names":["nikolasent/personmask_tflite"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/NikolasEnt/PersonMask_TFLite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolasEnt%2FPersonMask_TFLite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolasEnt%2FPersonMask_TFLite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolasEnt%2FPersonMask_TFLite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolasEnt%2FPersonMask_TFLite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NikolasEnt","download_url":"https://codeload.github.com/NikolasEnt/PersonMask_TFLite/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikolasEnt%2FPersonMask_TFLite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273953497,"owners_count":25197316,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-06T02:00:13.247Z","response_time":2576,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c-plus-plus","deep-learning","mobilenetv3","portrait-segmentation","semantic-segmentation","tensorflow-lite","tflite","v4l2","v4l2loopback","video","video-calls","video-processing"],"created_at":"2024-10-03T12:36:00.881Z","updated_at":"2025-09-06T19:32:31.083Z","avatar_url":"https://github.com/NikolasEnt.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Person Segmentation\n\nThe goal of the project is to inference a deep learning semantic segmentation model on a webcam video stream in Linux in order to remove background and provide a clean video stream with a person only. The need for such a project was imposed by the lack of such tools in Linux and the popularity of video calls services in present times.\n\n![Demo of the project output](readme_imgs/demo.gif \"Project demo\")\n\n*The app output demo. Based on a video from [vimeo](https://vimeo.com/14802778), which is licenced under CC Attribution license.*\n\nThe main feature of the project in terms of computer vision is the usage of a lightweight neural network model which is able to run on CPU in real-time, which is quite challenging for the semantic segmentation task. The project is implemented in C++17.\n\n*Note:* The project is in early alpha stage, so, it may contain bugs, and it is unreliable.\n\n- [Person Segmentation](#person-segmentation)\n  - [Dependances](#dependances)\n      - [GCC](#gcc)\n      - [Build system](#build-system)\n      - [FFmpeg](#ffmpeg)\n      - [OpenCV \u003e= 4.3](#opencv--43)\n      - [TensorFlow Lite](#tensorflow-lite)\n      - [v4l2loopback](#v4l2loopback)\n  - [How to build](#how-to-build)\n  - [How to run](#how-to-run)\n  - [All args](#all-args)\n  - [Project structure](#project-structure)\n  - [Performance](#performance)\n\n## Dependances\n\nAll dependences are downloaded in `~/dependencies/`\n\n#### GCC\n\nThe project was built and tested with gcc 9.3.0 and 10.1.1.\n\nGcc 9.3.0 can be installed and selected as default in Ubuntu by:\n\n```\nsudo apt-get update -y \u0026\u0026 \nsudo apt-get upgrade -y \u0026\u0026 \nsudo apt-get dist-upgrade -y \u0026\u0026 \nsudo apt-get install build-essential software-properties-common -y \u0026\u0026 \nsudo add-apt-repository ppa:ubuntu-toolchain-r/test -y \u0026\u0026 \nsudo apt-get update -y \u0026\u0026\nsudo apt-get install gcc-9 g++-9 -y \u0026\u0026 \nsudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-9 \u0026\u0026 \nsudo update-alternatives --config gcc\n```\n\n#### Build system\n\nThe program requires `cmake` (\u003e=3.11.3) and `make` to build.\n\n#### FFmpeg\n\nIf you'd like to input video from a video file, you'll need to compile FFmpeg. See instructions [here](http://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu). Actually, it needs libx264 only, so, minimal build and installation of dependencies on Ubuntu may look like:\n\n```\nsudo apt-get update -qq \u0026\u0026 sudo apt-get -y install \\\n  autoconf \\\n  automake \\\n  build-essential \\\n  cmake \\\n  git-core \\\n  libass-dev \\\n  libfreetype6-dev \\\n  libgnutls28-dev \\\n  libsdl2-dev \\\n  libtool \\\n  libva-dev \\\n  libvdpau-dev \\\n  libvorbis-dev \\\n  libxcb1-dev \\\n  libxcb-shm0-dev \\\n  libxcb-xfixes0-dev \\\n  pkg-config \\\n  texinfo \\\n  wget \\\n  yasm \\\n  nasm \\\n  zlib1g-dev \\\n  libx264-dev\n\n\nmkdir ~/dependencies/ffmpeg_sources\nmkdir ~/dependencies/ffmpeg_build\nmkdir ~/dependencies/bin\ncd ~/dependencies/ffmpeg_sources\nwget -O ffmpeg-snapshot.tar.bz2 https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2\ntar xjvf ffmpeg-snapshot.tar.bz2 \u0026\u0026 cd ffmpeg\n\nPATH=\"$HOME/dependencies/bin:$PATH\" PKG_CONFIG_PATH=\"$HOME/dependencies/ffmpeg_build/lib/pkgconfig\" ./configure  --prefix=\"$HOME/dependencies/ffmpeg_build\"   --pkg-config-flags=\"--static\"   --extra-cflags=\"-I$HOME/dependencies/ffmpeg_build/include\"   --extra-ldflags=\"-L$HOME/dependencies/ffmpeg_build/lib\"   --extra-libs=-lpthread   --extra-libs=-lm   --bindir=\"$HOME/dependencies/bin\"   --enable-gpl     --enable-libfreetype --enable-libx264 --enable-nonfree\nPATH=\"$HOME/dependencies/bin:$PATH\" make -j$(nproc) \u0026\u0026 make install\n```\n\nBe patient, it may take a while.\n\n#### OpenCV \u003e= 4.3\n\nInstullation [guide](https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html).\nBasic installation may look like:\n\n```\nmkdir ~/dependencies/opencv\ncd ~/dependencies\nwget https://github.com/opencv/opencv/archive/4.3.0.zip\nunzip 4.3.0.zip\ncd ~/dependencies/opencv-4.3.0\nmkdir build \u0026\u0026 cd build\nexport LD_LIBRARY_PATH=$HOME/dependencies/ffmpeg_build/lib/:$LD_LIBRARY_PATH\nexport PKG_CONFIG_PATH=$HOME/dependencies/ffmpeg_build/lib/pkgconfig:$PKG_CONFIG_PATH:\nexport PKG_CONFIG_LIBDIR=$HOME/dependencies/ffmpeg_build/lib/:$PKG_CONFIG_LIBDIR\n\nsudo apt-get install -y libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev\n\ncmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=$HOME/dependencies/opencv/ \\\n-D INSTALL_PYTHON_EXAMPLES=OFF  -D INSTALL_C_EXAMPLES=OFF -D WITH_FFMPEG=1 \\\n-D BUILD_LIST=core,imgproc,imgcodecs,highgui,video,videoio ..\nmake -j$(nproc)\nmake install\n```\n\nThe openCV should be compiled after ffmpeg in order to be build with it for ffmpeg backend support.\n\n#### TensorFlow Lite\n\nThe current implementation was tested with TFLite v. 2.2\n\n```\ncd ~/dependencies\ngit clone --branch r2.2 --single-branch  https://github.com/tensorflow/tensorflow.git\ncd tensorflow\n./tensorflow/lite/tools/make/download_dependencies.sh\n./tensorflow/lite/tools/make/build_lib.sh\n```\n\nDo not forget to update pathes of the libs in [CMakeLists.txt](CMakeLists.txt) (e.g., set TENSORFLOW variable) if you modified installation pathes.\n\nThe project uses deep learning model MobeleNetV3-segm from an open source [repo](https://github.com/OniroAI/Semantic-segmentation-with-MobileNetV3).\nIt was retrained for 256x256 px input resolution, and extra data was utilized for training.\n\n#### v4l2loopback\n\nVirtual video devices Linux kernel module is required to enable the program video output to be accessible from other programs.\n\nSome Linux distros have it already preinstalled, or one can install it by following [documentation](https://github.com/umlaeute/v4l2loopback).\n\nIt is available via apt for Ubuntu:\n\n```\nsudo apt-get install -y v4l2loopback-dkms\n```\n\n## How to build\n\n```\nmkdir build\ncd build\ncmake ..\nmake -j$(nproc)\n```\n\n## How to run \n\nFor test purposes, a test video is provided in `data/test.mp4`. The video is a random video from [vimeo](https://vimeo.com/14802778) with several speaking people in front of a camera, and it is licenced under CC Attribution license.\n\nBefore running the program, you have to create a virtual webcam with v4l2loopback for virtual system device output (you may skip the step if a debug visualization is required only):\n\n```\nsudo modprobe v4l2loopback devices=1 max_buffers=2 exclusive_caps=1 card_label=\"PersonMaskCam\"\n```\n\nNote a video device name it creates, like  `/dev/videoX`. You may use the camera as input device for video calls apps.\n\nKeep in mind, that in case you'd like to use different video resolution, you may have to restart the virtual video device.\n\nTo test it on a video file, just run\n```\ncd build\n./person_mask -i ../data/test.mp4 -d\n```\n\nIt will enable a debug visualization and run the program with a video file `../data/test.mp4` as input. Press ESC utton to stop the program.\n\nYou can redirect the output to a v4l2loopback virtual device by adding `-o /dev/videoX`, where /dev/videoX - a device created during the virtual video devices kernel module initialization.\n\n## All args\n```\n  -i, --input arg   input camera or path to input video file. (default:\n                    /dev/video0)\n  -o, --output arg  Output virtual camera device, like /dev/video1, do not\n                    provide the parameter to skip the results outputting to\n                    virtual video device.\n  -d, --debug       Draw a debug visualization with OpenCV. It also prints out\n                    frame processing time into stdout.\n  -h, --help        Print help\n```\n\n## Project structure\n\nThe main loop is defined in [src/main.cpp](src/main.cpp) file. It parses user input, initializes input virtual camera (frame grabbing from a video file or a webcam), applies predictor to the grabbed images and draws the results visualization.\n\nThe virtual camera [src/camera.cpp](src/camera.cpp) grab frames and put them into custom Buffer (see [include/buffer.h](include/buffer.h)).\n\nFrames from the frame buffer are processed with a predictor [src/predictor.cpp](src/predictor.cpp), which run a TFLite model and all necessary image and predicted mask processings. The model parameters (like the path to the model weights, segmentation threshold and a number of used CPU cores) could be adjusted in the [src/predictor.cpp](src/predictor.cpp) file.\n\nTo output results as a system virtual camera, a video v4l2-loopback video  writer is created with a function from [src/video_writer.cpp](src/video_writer.cpp).\n\n## Performance\n\nIt is able to run at ~50 fps at half of CPU cores and ~30 fps in single core mode on an Intel i5-8250U.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikolasent%2Fpersonmask_tflite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnikolasent%2Fpersonmask_tflite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikolasent%2Fpersonmask_tflite/lists"}