{"id":20373200,"url":"https://github.com/st235/hse.moderntoolsforsolvingcomputervisionproblems","last_synced_at":"2025-03-04T20:44:19.254Z","repository":{"id":213020780,"uuid":"732632686","full_name":"st235/HSE.ModernToolsForSolvingComputerVisionProblems","owner":"st235","description":null,"archived":false,"fork":false,"pushed_at":"2023-12-22T21:51:46.000Z","size":67276,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-15T06:49:43.400Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/st235.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-17T10:40:25.000Z","updated_at":"2023-12-18T00:23:49.000Z","dependencies_parsed_at":"2025-01-15T05:55:26.221Z","dependency_job_id":null,"html_url":"https://github.com/st235/HSE.ModernToolsForSolvingComputerVisionProblems","commit_stats":null,"previous_names":["st235/hse.moderntoolsforsolvingcomputervisionproblems"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.ModernToolsForSolvingComputerVisionProblems","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.ModernToolsForSolvingComputerVisionProblems/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.ModernToolsForSolvingComputerVisionProblems/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/st235%2FHSE.ModernToolsForSolvingComputerVisionProblems/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/st235","download_url":"https://codeload.github.com/st235/HSE.ModernToolsForSolvingComputerVisionProblems/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241921836,"owners_count":20042763,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T01:17:06.364Z","updated_at":"2025-03-04T20:44:19.187Z","avatar_url":"https://github.com/st235.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Original model\r\n\r\n[The original model](https://developers.google.com/mediapipe/solutions/vision/image_segmenter#selfie-model) performs selfie\r\nsegmentation.\r\n\r\nThe model is an exported **TensorFlow Lite** model. The features of the model are:\r\n- **Input**: the model expects **normalised** 256x256 RGB image.\r\n- **Recognises 6 classes**: background, hair, body-skin, face-skin, clothes, and accessories.\r\n- **Output**: a tensor of probabilities of size _256x256x6_. \r\n\r\nYou can get an idea of the segmentation masks for every class from the images below:\r\n\r\n| Original                                | Background                                  | Hair                            |\r\n|-----------------------------------------|---------------------------------------------|---------------------------------|\r\n| ![Original](./docs/model_original.jpeg) | ![Background](./docs/model_background.jpeg) | ![Hair](./docs/model_hair.jpeg) |\r\n\r\n| Body Skin                                 | Face Skin                                 | Clothes                               | Accessories                                   |\r\n|-------------------------------------------|-------------------------------------------|---------------------------------------|-----------------------------------------------|\r\n| ![Body Skin](./docs/model_body_skin.jpeg) | ![Face Skin](./docs/model_face_skin.jpeg) | ![Clothes](./docs/model_clothes.jpeg) | ![Accessories](./docs/model_accessories.jpeg) |\r\n\r\nThe original model can be found under [`original_model` folder](./original_model).\r\n\r\n# Working with TVM\r\n\r\nThe model has been converted to TVM IR, then tuned, compiled for `x86_64-linux-gnu` and exported.\r\n`x86_64-linux-gnu` is a target for the linux image that runs `python:3.9-slim-bullseye` Docker container.\r\n\r\nThe entire history of tuning is listed in [`TFLite_Converation.ipynb` notebook file](./tools/TFLite_Convertation.ipynb),\r\nthat has been used at the preparation step. \r\n\r\n## Tuning\r\n\r\nThe key parameters for tuning are given below:\r\n\r\n| Parameter      | Value | Notes                                                                                                         |\r\n|----------------|-------|---------------------------------------------------------------------------------------------------------------|\r\n| tuner          | xgb   | XGBoost works really good and much faster than GridSearch.                                                    |\r\n| early_stopping | False | Instead of stopping earlier I limit trials.                                                                   |\r\n| n_trial        | 333   | Empirically tuned value: not big to finish within a few hours, not small to actually find optimal parameters. |\r\n\r\nTuning helped and optimised the model for the specific hardware. I was able to spare about **350 ms** on average.\r\nYou can find the final optimisation statistic in [`tvm_model/tuning_statistics.log`](./tools/tuning_statistics.log) and\r\nthe evaluation summary from **Collab** below:\r\n\r\n![Tuning results](./docs/tuning_results.png)\r\n\r\n## Exporting the artifacts\r\n\r\nI considered multiple options before exporting the model: \r\n1. Using TVMC as a standalone archive\r\n2. Using build-in TVM methods to get a dynamic library\r\n\r\n### TVMC Package\r\n\r\nTVMC packs model in a really handy archive. Though there is a wee issue with TVMC: \r\nthe parameters are not baked with the model, and you need to provide them separately. \r\nIt seemed it was necessary to serialise them and keep separately. So to work with a model with weights one needs to\r\nprovide 2 files per model: model archive and the parameters.\r\n\r\nI tried to export files and work with them (you can find the code in the notebook linked above).\r\nExported model lays under [`tvm_model` folder](./tvm_model) and is called `tvm_not_tuned_selfie_multiclass.tar`.\r\nThe file with parameters can be found under the same folder and is called [`tvmc_serialised_parameters.npy`](./tvm_model/tvmc_serialised_parameters.npy).\r\n\r\n### TVM built-in methods\r\n\r\nThe pair [`lib.export_library`](https://tvm.apache.org/docs/reference/api/python/runtime.html?highlight=export_library#tvm.runtime.Module.export_library) -\r\n[`tvm.runtime.load_module`](https://tvm.apache.org/docs/reference/api/python/runtime.html?highlight=runtime#tvm.runtime.load_module) can export and then load the model\r\nas a dynamic library specifically compiled for the hardware.\r\n\r\nI preferred this way over **TVMC** as it allows to bake-in the parameters of the model and provide them within the same\r\nfile. This approach is used in the final solution.\r\n\r\n### Tuning script\r\n\r\n_⚠️ Please, do use **Python notebook** as your primary choice and use the script only as a last resort option._\r\n\r\n[The script](./tools/tflite_model_converter.py) contains necessary logic to convert and tune **TFLite** model using TVM.\r\n\r\nThe script has a few arguments, described below:\r\n\r\n| Argument                               | Type        | Notes                                        |\r\n|----------------------------------------|-------------|----------------------------------------------|\r\n| Positional argument, always goes first | str         | Path to the TFLite model                     |\r\n| input_shape                            | list of int | Shape of the input layer of the model        |\r\n| input_tensor                           | str         | Name of the input tensor, `input` by default |\r\n| input_dtype                            | str         | Type of the input data, `float32` by default |\r\n| target                                 | str         | Compilation and optimisation target          |\r\n| run_tuning                             | bool        | Specify, to also run tuning                  |\r\n\r\nTo use it without fine-tuning you can use the command below:\r\n\r\n```bash\r\npython tflite_model_converter.py ../original_model/selfie_multiclass_256x256.tflite --input_tensor input_29 -is 1 256 256 3 --target llvm\r\n```\r\n\r\nTo run it with fine-tuning you need to slightly change the command:\r\n\r\n```bash\r\npython tflite_model_converter.py ../original_model/selfie_multiclass_256x256.tflite --input_tensor input_29 -is 1 256 256 3 --target llvm --run_tuning\r\n```\r\n\r\n# Bot\r\n\r\n## How to use this bot \r\n\r\nThe bot is really easy to control and interact with. It supports 3 commands:\r\n1. `/segmentation`\r\n2. `/change-hair-color`\r\n3. `/change-background`\r\n\r\nWhen the command is executed it returns an image or series of images and performance information.\r\nPerformance compares **original** and **tuned** model and is similar to the example below:\r\n\r\n```text\r\nInference of default model took 264 ms.\r\nInference of tuned model took 185 ms.\r\n```\r\n\r\nLet's take a closer look at these commands.\r\n\r\n### Segmentation\r\n\r\n![Bot segmentation](docs/bot_segmentation.png)\r\n\r\nIn the segmentation mode you need to send a selfie in a **quick send** mode.\r\n\r\nThe result is 7 images: 6 corresponding to the classes masks and the original image.\r\n\r\n### Change Hair Color\r\n\r\n![Bot change hair color](docs/bot_change_color_hair.png)\r\n\r\nReceives a selfie and changes the color of the person's hair.\r\n\r\nThis command is a two steps command:\r\n1. Waits for a selfie image\r\n2. Asks to send a new color value as a hexadecimal rgb string\r\n\r\nThe result is a modified rgb image with new hair color.\r\n\r\n### Change Background\r\n\r\n![Bot change background](docs/bot_change_background.png)\r\n\r\nChanges background in selfie, similar to what modern video call apps are doing.\r\n\r\nThis command also works in two steps:\r\n1. Waits for a selfie image\r\n2. Asks to provide a new background\r\n\r\nThe command returns a modified selfie with replace background.\r\n\r\n## Deployment\r\n\r\nAnd the last but not least, the instructions how to deploy the bot.\r\nThe bot is using `docker-compose`.\r\n\r\n### Create an environment file with a token\r\n\r\nFirst of all, it is required to create an `.env` file and declare `TELEGRAM_BOT_TOKEN` variable in there.\r\nThe content of a file should look similar to the snippet below:\r\n\r\n```bash\r\nTELEGRAM_BOT_TOKEN=0000000000:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\r\n```\r\n\r\nThe real token can be obtained from the [`@botfather`](https://t.me/botfather).\r\n\r\n#### Build docker-compose image\r\n\r\nRun the command below to build the docker image.\r\n\r\n`docker-compose --project-name hse_tvm_bot build`\r\n\r\n#### Run with environment\r\n\r\nTo run the image one should use the command below.\r\n\r\n`docker-compose --project-name hse_tvm_bot --env-file .env up`\r\n\r\nAfter running the command, the bot will go online and it would be possible to use it from Telegram.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fst235%2Fhse.moderntoolsforsolvingcomputervisionproblems","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fst235%2Fhse.moderntoolsforsolvingcomputervisionproblems","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fst235%2Fhse.moderntoolsforsolvingcomputervisionproblems/lists"}