{"id":25531937,"url":"https://github.com/generousman/nonlinearnavigation","last_synced_at":"2025-04-11T11:35:26.004Z","repository":{"id":62025226,"uuid":"551987659","full_name":"GenerousMan/NonLinearNavigation","owner":"GenerousMan","description":"Code of paper 《Automatic Generation of Interactive Nonlinear Video for Online Apparel Shopping Navigation》.","archived":false,"fork":false,"pushed_at":"2022-10-27T08:04:30.000Z","size":32369,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"init","last_synced_at":"2025-03-25T08:01:29.354Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GenerousMan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-15T14:56:19.000Z","updated_at":"2022-11-01T02:50:50.000Z","dependencies_parsed_at":"2023-01-20T17:46:49.860Z","dependency_job_id":null,"html_url":"https://github.com/GenerousMan/NonLinearNavigation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GenerousMan%2FNonLinearNavigation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GenerousMan%2FNonLinearNavigation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GenerousMan%2FNonLinearNavigation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GenerousMan%2FNonLinearNavigation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GenerousMan","download_url":"https://codeload.github.com/GenerousMan/NonLinearNavigation/tar.gz/refs/heads/init","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248384575,"owners_count":21094743,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-20T01:42:29.461Z","updated_at":"2025-04-11T11:35:25.976Z","avatar_url":"https://github.com/GenerousMan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NonLinear Interactive Video For Shopping Navigation\nCode of paper 《Automatic Generation of Interactive Nonlinear Video for Online Apparel Shopping Navigation》.\n\n\u003cp align='center'\u003e\n    \u003cimg src=\"imgs/pipeline.png\", width=\"800\"\u003e\n\u003c/p\u003e\n\n**Notably, using interactive video to implement shopping navigation is a new research topic, and our solution is of course not perfect. Please refer to our failure cases and limitations before using this repo.**\n\n## Update\n- [2022-10-27] Add example files, in examples/.\n- [2022-10-25] Update README, to introduce out work.\n- [2022-10-19] Create the project, and upload main codes of this paper.\n\n## Abstract\n\nWe present an automatic generation pipeline of interactive nonlinear video for online apparel shopping navigation. Our approach is inspired by Google’s \"Messy Middle\" theory, which suggests that people mentally have two tasks of exploration and evaluation before purchasing. Given a set of apparel product presentation videos, our navigation UI will organize these videos for users’ product exploration and automatically generate interactive videos for user product evaluation. To support automatic methods, we propose a video clustering similarity (CSIM) and a camera movement similarity (MSIM), as well as a comparative video generation algorithm for product recommendation, presentation, and comparison. To evaluate our pipeline’s effectiveness, we conducted several user studies. The results show that our pipeline can help users complete the consumption process more efficiently, making it easier for users to understand and choose the product.\n\n\n## Theoretical support —— Messy Middle\n\u003cp align='center'\u003e\n    \u003cimg src=\"imgs/messy_middle.jpeg\", width=\"500\"\u003e\n\u003c/p\u003e\n    Prior work has studied the behavioral logic of consumers and a \"messy middle\" theory was proposed, which noted that consumers often wander in the two states of exploration and evaluation when shopping online. Consumers explore their options and expand their consideration sets; then – either sequentially or simultaneously – they evaluate the options and narrow down their choices. Existing online shopping methods need to constantly switch pages to view and compare products. Such a shopping method reduces the exploration and evaluation efficiency and increases the time for customers to make a decision.\n\n    To shorten the shopping time between product exploration and decision-making, we propose an automatic approach for generating nonlinear videos into two-level, coarse-level exploration and fine-level evaluation, in support of online clothing shopping navigation. Our approach can automatically generate interactive nonlinear videos for product presentation and comparison based on consumers’ interactions.\n\n\n## Requirements\n\n``` bash\npip install requirements.txt\n```\n\n## Weights\n\n- [\\[Attribute and Category\\]](https://drive.google.com/drive/folders/1vfO2GXi0wsJQ6zvPRSKOIMYLLAAvncuG?usp=sharing) Put in libs/Fashion/checkpoints/, to predict the attributes and categories of apparel products in videos.\n- [\\[Unet\\]](https://drive.google.com/drive/folders/1lssxarbbnPggT94pEnwqnVPSGpNdFAo1?usp=sharing) Put in libs/unet_segmentation/, to segment the apparel products in videos.\n- [\\[Detail Classification\\]](https://drive.google.com/drive/folders/1QgcrP0ZMKDTb8A2tHxonM_3mys-jkMVO?usp=sharing) Put in libs/Detail/models/, to classify the detail shots.\n- [\\[Landmark\\]](https://drive.google.com/drive/folders/1BI7GLUjwdPguVpxneGPQK8dnPXkM_sBi?usp=sharing) Put in libs/Landmark/models/, to predict the clothing landmarks in full-shots.\n- [\\[MaskROI\\]](https://drive.google.com/drive/folders/1WPAnizJJ_tfr1Q5uESxzy3a_L0HPdK-B?usp=sharing) Put in libs/maskRoi/weights/, to predict the ROI of human in frames.\n- [\\[AlphaPose\\]](https://drive.google.com/drive/folders/1Fi_jvlc3kZUwwi6d6xikgd9ZGePTpYEh?usp=sharing) Put in libs/AlphaPose/models/, to detect the keypoints of people in frames.\n- [\\[ML Models\\]](https://drive.google.com/drive/folders/1SZ3kZFp6NJGVMQkpYnc1Y_YcEjuRHqQu?usp=sharing) Put in model/, to obtain advantage features, such as view, direction of people in frame.\n\n## Usage - Video Association Algorithm\n\nIn the video association algorithm, we sample the input product video at intervals of $t$ frames and extract the feature $F=\\{F^{cate}, F^{attr}, F^{color}\\}$. We construct a graph through the average of all features $F$ and use this graph to show the associations of categories, attributes, and colors between clothing products. Video association algorithms can be divided into feature extraction and graph construction.\n\n\u003cp align='center'\u003e\n    \u003cimg src=\"imgs/exploration.jpg\", width=\"800\"\u003e\n\u003c/p\u003e\n\nTo recommend similar products (in many product videos), you can use this command:\n\n``` bash\npython examples/example_recommend.py\n```\n\nIt may takes long time to extract features and build the products' graph. After calculating all features of all videos, we will save them in a .pickle file. If there are lots of video nodes, it will also take a lot of space to save the graph.\n\n\n## Usage - Shot Association Algorithm\n\nIn the shot association algorithm, we automatically attach detail shots to the full shot. When consumers evaluate the product, they can click the area of interest in the video to obtain more targeted information. The algorithm can be divided into video shot classification, detailed shot classification, and keypoint detection.\n\n\u003cp align='center'\u003e\n    \u003cimg src=\"imgs/evaluation.png\", width=\"800\"\u003e\n\u003c/p\u003e\n\n\nTo associate close-up shots and full-shot, you can use this command:\n\n``` bash\npython examples/example_single_presentation.py\n```\n\nThis example will only generate the keypoints' position in the full-shot, and the classification results of close-up. These results will be saved as a .yaml file. Through our player, we can play this .yaml file as an interactive video.\n\n\n\n## Usage - Comparative Video Generation Algorithm\n\nThe comparison algorithm we propose is designed to help consumers compare multiple interested products simultaneously. When comparing products, consumers can select the view (such as whole, medium, close-up) and direction (such as left, right, front, back) of the clothing in the video. Based on shots filtered by these tags, we train a neural network to generate a comparative video according to the optical flow features of camera movement. The comparative video generation algorithm can be divided into shot label extraction, optical flow feature calculation, and shot sequence generation.\n\n\u003cp align='center'\u003e\n    \u003cimg src=\"imgs/optical_CNN.jpg\", width=\"800\"\u003e\n\u003c/p\u003e\n\nTo generate comparative shot sequence, you can use this command:\n\n``` bash\ncd examples/example_compare/\njupyter notebook\n```\n\nWe prepare four examples in the folder.\n\n- [\\[example_raft.ipynb\\]](examples/example_compare/example_raft.ipynb): The example of using raft to get optical features.\n\n- [\\[example_optical_flow.ipynb\\]](examples/example_compare/example_optical_flow.ipynb): The example of using opencv to get optical features.\n\n- [\\[example_train_editor.ipynb\\]](examples/example_compare/example_train_editor.ipynb): The example of training our DCDP-CNN.\n\n- [\\[example_cut_editor.ipynb\\]](examples/example_compare/example_cut_editor.ipynb): The example of using DCDP-CNN to generate shot sequences.\n\n## What's More\n\nWe also provide many tools and examples, to help you use any unit of our methods, such as alphapose, yolo, clothing segmentation, clothing landmarks detection.\n\nIn `examples/` and `tools/`, we hope you can find useful code for your work.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenerousman%2Fnonlinearnavigation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgenerousman%2Fnonlinearnavigation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenerousman%2Fnonlinearnavigation/lists"}