{"id":18811422,"url":"https://github.com/enot-autodl/enotdatasphere","last_synced_at":"2026-01-11T07:30:14.530Z","repository":{"id":55981316,"uuid":"317817578","full_name":"ENOT-AutoDL/ENOTDataSphere","owner":"ENOT-AutoDL","description":"This is a repository with examples of usage ENOT framework in Yandex DataSphere","archived":false,"fork":false,"pushed_at":"2020-12-03T07:44:13.000Z","size":21298,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-29T23:55:01.062Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ENOT-AutoDL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-02T09:51:50.000Z","updated_at":"2021-01-20T09:18:02.000Z","dependencies_parsed_at":"2022-08-15T10:40:57.419Z","dependency_job_id":null,"html_url":"https://github.com/ENOT-AutoDL/ENOTDataSphere","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ENOT-AutoDL%2FENOTDataSphere","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ENOT-AutoDL%2FENOTDataSphere/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ENOT-AutoDL%2FENOTDataSphere/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ENOT-AutoDL%2FENOTDataSphere/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ENOT-AutoDL","download_url":"https://codeload.github.com/ENOT-AutoDL/ENOTDataSphere/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239748254,"owners_count":19690232,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T23:26:15.258Z","updated_at":"2026-01-11T07:30:14.456Z","avatar_url":"https://github.com/ENOT-AutoDL.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ENOTDataSphere\nThis is a repository with ENOT framework usage examples in Yandex DataSphere.\n\n## Install\n* Go to \"install\" folder;\n* Open Install.ipynb ;\n* Execute all the cells one by one;\n* Now you can use all Jupyter Notebooks from this repository;\n\n\n## To reproduce results from the webinar:\n* Create your own project in Yandex DataSphere - https://cloud.yandex.ru/services/datasphere \n* Clone this repository into your DataSphere project (Git -\u003e Clone);\n* Download models from the link below, unzip file(Snippets -\u003e Extract ZIP file.py) \\\nand put it in the root directory of the project(so the path to models should be './ENOTDataSphere/models/base_model.pth'): \nhttps://yadi.sk/d/nbaV1N1tQMSPpg\n* Download test dataset from the link below, unzip it(Snippets -\u003e Extract ZIP file.py) \\\nand create \"data\" folder in \"mmdet_tools\" folder and copy unziped folder('masks') in './ENOTDataSphere/mmdet_tools/data/'(so the path to your data should be './ENOTDataSphere/mmdet_tools/data/'):\nhttps://yadi.sk/d/MwN9o5LmLi5Cvg\n* Download test video from this link, create folder \"video\" and put downloaded video there:\nhttps://yadi.sk/i/hE05IF9-OEwvKg\n\n##### Now you can use all notebooks in this repository\n* demo_video.ipynb - demo of masks detection on video. You can download your own video, change path to the video and detect masks;\n* pretrain_search.ipynb - in this notebook you can find examples of enot_pretrain, enot_search phases and get test metrics on the dataset;\n* train_baseline.ipynb - example of train baseline model with MMDetection;\n\n\n## Train detector on custom dataset with MMDET+ENOT\n### To train your custom dataset for detection and use optimization framework ENOT you should:\n#### 1 - Prepare your dataset to COCO annotation format. \n* You should have 3 .json files, like *. train.json, test.json, val.json .* and 3 folders(train, test, val) with images. Make a folder(like 'my_dataset_name') and copy all these files into it. About COCO annotation format you can read here - https://mmdetection.readthedocs.io/en/latest/2_new_data_model.html\n* In 'mmdet_tools' make a folder 'data' and copy your directory  with dataset('my_dataset_name') to 'data'.\n#### 2 - Prepare configs for ENOT_Pretrain, ENOT_search and tune phases.\n* In 'mmdet_tools/configs/_base_/datasets' in files 'mask_face.py' and 'mask_face_pretrain.py' set to variable 'data_root' path to your dataset.\nSet 'ann_file' parameter for train, test and val dictionaries with paths to yours .json files(train.json, test.json, val.json)\n* If you want to change searchspace you should do it in 'mmdet_tools/configs/_base_/models/search_space_ssd_masks.py'\n* For enot_pretrain phase you should change paths to your dataset in 'mmdet_tools/configs/wider_face/search_space_ssd_masks.py', also you can set here augmentations you want.\n* For enot_search phase you should change paths to your dataset in 'mmdet_tools/configs/wider_face/search_space_ssd_masks_search.py', dataset and augmentations should be the same like in pretrain phase!\n###### Detailed information about MMDetection config system you can find here - https://mmdetection.readthedocs.io/en/latest/tutorials/config.html\n\n#### 3 - run enot_pretrain, enot_search phases\nTo start enot_pretrain you should:\n* Prepare config(see 2);\n* In 'mmdet_tools/run_enot_pretrain.py' in pretrain_cfg dictionary you should change: 'epochs' - number of epochs, 'mmdet_config_path' - path to 'mmdet_tools/configs/_base_/models/search_space_ssd_masks.py', 'experiment_dir' - path to save checkpoints. In 'enot_pretrain' function you can change type of optimizer, learning rate, set scheduler, and batch size;\n* When all configs, learning procedure and paths in 'mmdet_tools/run_enot_pretrain.py' are ready, from jupyter notebook just call 'run_enot_pretrain' function from 'mmdet_tools/run_enot_pretrain.py';\n* Prepare config(see 2);\n* Choose best checkpoint from pretrain phase;\n* In 'mmdet_tools/run_enot_search.py' in pretrain_cfg dictionary you should change: 'epochs' - number of epochs, 'mmdet_config_path' - path to 'mmdet_tools/configs/_base_/models/search_space_ssd_masks.py', 'experiment_dir' - path to save checkpoints, 'pretrain_checkpoint_path' - path to best checkpoint from pretrain phase. In 'enot_search' function you can change type of optimizer, learning rate, set scheduler, and batch size;\n* When all configs, learning procedure and paths in 'mmdet_tools/run_enot_search.py' are ready, from jupyter notebook just call 'run_enot_search' function from 'mmdet_tools/run_enot_search.py'. If you can set up parameter 'latency_loss_weight' to vary complexity of model to find, bigger 'latency_loss_weight' - more lightweight model you will find;\n\n#### 4 - run tune found model\n* Prepare config 'mmdet_tools/configs/_base_/models/mobilenet_from_search_space_mask.py', set indices of best models from search phase. And 'mmdet_tools/configs/wider_face/mobilenet_from_search_space_mask.py' here you should set path to dataset, augmentations, optimizer, learning rate;\n* Export weights(.pth file) of the best model. Example you can see in 'mmdet_tools/build_searched_arch_ckpt.py';\n* Call train from 'mmdet_tools/tools/train_baseline.py'. Set parameter 'config' - path to 'mmdet_tools/configs/wider_face/mobilenet_from_search_space_mask.py', \n'full_pretrained_model' - path to exported .pth file;\n\n\n## Tutorials for ENOT framework you can find here:\nhttps://github.com/ENOT-AutoDL/ENOT_Tutorials\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fenot-autodl%2Fenotdatasphere","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fenot-autodl%2Fenotdatasphere","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fenot-autodl%2Fenotdatasphere/lists"}