{"id":20885667,"url":"https://github.com/nripstein/thesis-100-doh","last_synced_at":"2026-04-19T16:37:19.004Z","repository":{"id":221279330,"uuid":"753911866","full_name":"nripstein/Thesis-100-DOH","owner":"nripstein","description":"My modified version of 100-DOH for undergrad thesis. Adopted from https://github.com/ddshan/hand_object_detector","archived":false,"fork":false,"pushed_at":"2024-05-28T17:31:01.000Z","size":2804,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-19T10:45:12.446Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nripstein.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-07T02:44:25.000Z","updated_at":"2024-05-28T17:31:05.000Z","dependencies_parsed_at":"2024-04-08T21:51:09.878Z","dependency_job_id":"82c72da1-3324-4f62-b388-5a4458a121da","html_url":"https://github.com/nripstein/Thesis-100-DOH","commit_stats":null,"previous_names":["nripstein/thesis-100-doh"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nripstein%2FThesis-100-DOH","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nripstein%2FThesis-100-DOH/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nripstein%2FThesis-100-DOH/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nripstein%2FThesis-100-DOH/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nripstein","download_url":"https://codeload.github.com/nripstein/Thesis-100-DOH/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243268087,"owners_count":20263803,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T08:14:09.129Z","updated_at":"2025-12-26T17:02:42.140Z","avatar_url":"https://github.com/nripstein.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Contact Duration Detection\nFor my senior honours thesis in the \u003ca href=\"https://pnb.mcmaster.ca/goldreich-lab/CurrentRes.html#Modeling\"\u003eGoldreich Lab\u003c/a\u003e, we have collected 500+ hours of videos of participants' hands as they perform our haptic categorization experiment. My aim is to automatically detect how long participants hold objects in these videos.\n### Goal\n![repo_goal](assets/goal.png)\n\n### Preliminary Results\n![prelim1](assets/prelim1.png)\n![prelim2](assets/prelim2.png)\n## Method\n\u003cb\u003eOverview of proposed technique\u003c/b\u003e\n\u003col\u003e\n\u003cli\u003eUse \u003ca href=\"https://github.com/ddshan/hand_object_detector\"\u003e100-DOH\u003c/a\u003e to assign frame-wise labels to video.\u003c/li\u003e\n\u003cli\u003eRefine predictions to minimize over-segmentation error.\u003c/li\u003e\n\n\u003cb\u003eSegment refining techniques\u003c/b\u003e\nSometimes the 100-DOH network detects the object in the hand of the experimenter, who wears blue latex gloves. These false positives are easy to eliminate: in each frame where a hand in contact with an object is detected, we identify the amount of blue inside the bounding box of the hand in contact with the object; if the bounding box contains mostly blue pixels, then we can be sure that the hand touching the object is wearing a latex glove and belongs to the experimenter. We can therefore conclude that the frame was initially a false positive.\n\nWe are still desciding on the best method to refine segmentations. Current ideas include:\n\u003cul\u003e\n\u003cli\u003eDual pass gap elimination. Example: fill in any gaps between predicted \"contact\" labels lasting fewer than 10 frames with \"contact\" labels. Subsequently, gaps between predicted \"non-contact\" labels lasting fewer than 5 frames are similarly filled.\u003c/li\u003e\n\u003cli\u003eSliding window voting: a sliding window could assign frame-wise labels using majority voting\u003c/li\u003e\n\n\u003c/ul\u003e\nAccording to \u003ca href=\"https://arxiv.org/abs/2210.10352\"\u003eDing et al.\u003c/a\u003e, the \u003ca href=\"https://arxiv.org/abs/2007.06866\"\u003eASRF\u003c/a\u003e network can refine temporal segmentations in a model-indipendent way by considering predicted labels, frame features and temporal context. We hope to investigate using it as part of our refinement sequence\n\n## Future directions\n\u003cb\u003eWeakly-Supervised Temporal Action Segmentation\u003c/b\u003e\nWe hope to the identify frames from our refined 100-DOH predictions which have very confident predicted labels. These labels can be fed into a temporal action segmentation (TAS) model which learns from time-stamp supervision. Time-stamp supervision involves labeling as few as a single frame from each action category in the training set. Performance is only a few years behind fully supervised TAS methods. The models from \u003ca href=\"https://arxiv.org/abs/2103.06669\"\u003eLi et al., 2021\u003c/a\u003e and \u003ca href=\"https://arxiv.org/abs/2212.11694\"\u003eDu et al., 2022\u003c/a\u003e serve as possible options for implimentation.\n\n\n\n\u003cb\u003eExtending our method for robust detection: self-supervised sister network\u003c/b\u003e\nWe hope to extend our method by adding a self-supervised object detector network which runs alongside the existing R-CNN detection network. This network would be taylor-made to recognize the gray coin-shaped objects from our experiment. \n\nThe current \u003ca href=\"https://github.com/ddshan/hand_object_detector\"\u003e100-DOH\u003c/a\u003e network is trained to recognize when hands are in contact with generic portable and stationary objects. It can often identify when participant hands are in contact with our object, but sometimes fails to, especially near the initiation of contact when the object is still on the table. We could extract object bounding boxes from 100-DOH, and use the results as training data for a supervised object detector. With this labeled data, this sister detector will be able to identify our object more accuratley than the current generic portable object detector. In this way, we could avoid a human in the loop, and end up with a detector tailor-made for our objects.  When the hand bounding box generated by the current network overlaps with the sister network's object bounding box, we can determine that the participant's hands are in contact with the object.\n\nThe automated nature of labeled data aquisition for the sister network aleviates concerns about overfitting: we can train the network on frames from every video it will eventually perform inference on because the network will not need to generalize to different videos, only different sections of video in the labeled training set.\n\n\n\n\n\u003cb\u003eComparisons to other techniques\u003c/b\u003e\nWe hope to attempt to compare our method a more conventional temporal action segmentation technique. One wich requires only timestamp supervision (infrequently labeled frames with associated timestamps) would be suitable for our task.\n## About this repository\nAdopted from [100-DOH Repository](https://github.com/ddshan/hand_object_detector), which is the code for the paper *Understanding Human Hands in Contact at Internet Scale* (CVPR 2020, **Oral**).\n\nDandan Shan, Jiaqi Geng*, Michelle Shu*, David F. Fouhey\n\n![method](assets/method.png)\n\nMore information can be found at the* [Project and dataset webpage](http://fouheylab.eecs.umich.edu/~dandans/projects/100DOH/)\n\n## Installation\nThis was only tested on Ubuntu 22.04 with an NVIDIA GPU. CUDA and cuDNN must be installed before using this repository.\n\n\n## Prerequisites\n\nCreate a conda environment, install pytorch-1.12.1, cuda-11.3:\n* python=3.8\n* cudatoolkit=11.3\n* pytorch=1.12.1\n\n\n\n## Preparation\n\nFirst of all, clone the code\n```\ngit clone https://github.com/nripstein/Thesis-100-DOH \u0026\u0026 cd hand_object_detector\n```\n\n## Environment \u0026 Compilation\n### Environment Setup:\nCopy and paste the following commands into the command line:\n```\nconda create --name handobj_new python=3.8\nconda activate handobj_new\nconda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch\npip install -r requirements.txt\ncd lib\n# then install gcc 10  \nsudo apt install gcc-10 g++-10\nexport CC=/usr/bin/gcc-10\nexport CXX=/usr/bin/g++-10\n# now that we have gcc 10, can compile\nCC=/usr/bin/gcc-10 CXX=/usr/bin/g++-10 python setup.py build develop\n```\nInstall all the python dependencies using pip:\n```\npip install -r requirements.txt\n```\n\n\u003c!-- You will meet some errors about coco dataset: (not the best but the easiest solution)\n```\ncd data\ngit clone https://github.com/pdollar/coco.git \ncd coco/PythonAPI\nmake\n``` --\u003e\n\u003c!-- \nIf you meet some error about spicy, make sure you downgrade to scipy=1.1.0:\n```\npip install scipy=1.1.0\n``` --\u003e\n\nPS:\n\nSince the repo is modified based on [faster-rcnn.pytorch](https://github.com/jwyang/faster-rcnn.pytorch/tree/pytorch-1.0) (use branch pytorch-1.0), if you have futher questions about the environments, the issues in that repo may help.\n\n\n## Demo\n\n### Image Demo\n\n**Download models** by using the links in the table above from google drive.\n\n\n**Save models** in the **models/** folder:\n```\nmkdir models\n\nmodels\n└── res101_handobj_100K\n    └── pascal_voc\n        └── faster_rcnn_{checksession}_{checkepoch}_{checkpoint}.pth\n```\n\n\n\n**Simple testing**: \n\nPut your images in the **images/** folder and run the command. A new folder **images_det** will be created with the visualization. Check more about argparse parameters in demo.py.\n```\nCUDA_VISIBLE_DEVICES=0 python demo.py --cuda --checkepoch=xxx --checkpoint=xxx\n```\n\n\n### One Image Demo Output:\n\nColor definitions:\n* yellow: object bbox\n* red: right hand bbox\n* blue: left hand bbox\n\nLabel definitions:\n* L: left hand\n* R: right hand\n* N: no contact\n* S: self contact\n* O: other person contact\n* P: portable object contact\n* F: stationary object contact (e.g.furniture)\n\n\n![demo_sample](assets/boardgame_848_sU8S98MT1Mo_00013957.png)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnripstein%2Fthesis-100-doh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnripstein%2Fthesis-100-doh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnripstein%2Fthesis-100-doh/lists"}