{"id":13435958,"url":"https://github.com/ehsanik/dogTorch","last_synced_at":"2025-03-18T12:30:35.522Z","repository":{"id":77425080,"uuid":"132189134","full_name":"ehsanik/dogTorch","owner":"ehsanik","description":"Who Let The Dogs Out? Modeling Dog Behavior From Visual Data https://arxiv.org/pdf/1803.10827.pdf","archived":false,"fork":false,"pushed_at":"2021-11-17T20:46:18.000Z","size":1726,"stargazers_count":76,"open_issues_count":0,"forks_count":9,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-10-27T18:59:58.974Z","etag":null,"topics":["computer-vision","dog","dog-movements","ego-centric-videos","imitation-learning","modeling-dog","recurrent-neural-networks","representation-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ehsanik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-04T21:06:56.000Z","updated_at":"2024-07-22T09:11:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"cb6b69f4-c960-4338-b286-86e39aaede3b","html_url":"https://github.com/ehsanik/dogTorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FdogTorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FdogTorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FdogTorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FdogTorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ehsanik","download_url":"https://codeload.github.com/ehsanik/dogTorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244222074,"owners_count":20418440,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","dog","dog-movements","ego-centric-videos","imitation-learning","modeling-dog","recurrent-neural-networks","representation-learning"],"created_at":"2024-07-31T03:00:41.328Z","updated_at":"2025-03-18T12:30:35.494Z","avatar_url":"https://github.com/ehsanik.png","language":"Python","funding_links":[],"categories":["Software"],"sub_categories":["Machine Learning"],"readme":"## [Who Let The Dogs Out? Modeling Dog Behavior From Visual Data](https://arxiv.org/pdf/1803.10827.pdf)\nBy Kiana Ehsani, Hessam Bagherinezhad, Joseph Redmon, Roozbeh Mottaghi, and Ali Farhadi\n\n### Abstract\n\nDogs are intelligent. Let's learn from them! We introduce DECADE, a dataset of ego-centric videos from a dog’s perspective as well as her corresponding movements.\nIn this paper we propose a model that can\n1. Learn to act like a dog (predicting the dog’s future moves),\n2. Learn to plan like a dog (estimating a sequence of movements that take the state of the dog’s world from what is observed at a given time to a desired observed state),\n3. Learn from a dog (exploring the potentials of using the dog's movements for representation learning).\n\nThis project is presented as poster in CVPR'18.\n\n\u003ccenter\u003e\u003cimg src=\"figs/teaser_attach.jpg\" height=\"250px\" \u003e\u003c/center\u003e\n\n### Citing This Work\n\nIf you find this project useful in your research, please consider citing:\n\n\t@inproceedings{ehsani2018dog,\n\t    author = {Ehsani, Kiana and Bagherinezhad, Hessam and Redmon, Joseph and Mottaghi, Roozbeh and Farhadi, Ali},\n\t    title = {Who Let The Dogs Out? Modeling Dog Behavior From Visual Data},\n\t    booktitle = {CVPR},\t    \n\t    year = {2018}\n\t}\n\n### Dataset\n\nWe introduce DECADE, a dataset of ego-centric dog\nvideo and joint movements. The dataset includes 380 video\nclips from a camera mounted on the dog’s head. It also includes\ncorresponding information about body position and\nmovement. Overall we have 24500 frames. We use 21000\nof them for training, 1500 for validation, and 2000 for testing.\nTrain, validation, and test splits consist of disjoint\nvideo clips.\n\nWe use a GoPro camera on the dog’s head to capture the\nego-centric videos and sub-sample frames at the rate of 5\nfps. An Arduino on the dog’s back connects to the IMUs and\nrecords the positional information. It also collects audio\ndata via a microphone mounted on the dog’s back. We synchronize\nthe GoPro with the IMU measurements using audio\ninformation.\n\n\u003ccenter\u003e\u003cimg src=\"figs/teaser_data_collect.jpg\" height=\"300px\" \u003e\u003c/center\u003e\n\nThe dataset can be downloaded from [here](https://drive.google.com/file/d/1ctoCEu7Oacn2LKrS2_AbZnrNXNixuvWb/view?usp=sharing).\n\n### Running the code\n\n#### Requirements\n\nThis source code is implemented using PyTorch framework (v.0.2.0_3). For installing the framework and its prerequisites visit [PyTorch official website](http://pytorch.org/).\n\n#### Installation\n \n1. Clone the repository using the command\n\n\t\tgit clone https://github.com/ehsanik/dogTorch\n\t\tcd dogTorch\n\n3. Download the dataset from [here](https://drive.google.com/file/d/1ctoCEu7Oacn2LKrS2_AbZnrNXNixuvWb/view?usp=sharing) and unzip it.\n2. Make a link to the dataset.\n\n\t\tln -s /PATH/TO/DATASET data\n\n3. Download the pretrained weights from [here](https://drive.google.com/file/d/1nqcmyuqhNERz9RVC7lQte2Hb2chxbw8c/view?usp=sharing) and unzip it.\n\n#### Acting like a dog\n\nWe predict how the dog acts in the visual world in response to various situations. Specifically, we model the future actions of the dog given a sequence of previously seen images. To reproduce the results in the paper:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth class=\"tg-us36\"\u003eModel\u003c/th\u003e\n\u003cth class=\"tg-us36\"\u003eTest Accuracy\u003c/th\u003e\n\u003cth class=\"tg-us36\"\u003ePerplexity\u003c/th\u003e\n\u003cth class=\"tg-us36\"\u003eAngular metric\u003c/th\u003e\n\u003cth class=\"tg-us36\"\u003eAll joints\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd class=\"tg-us36\"\u003eCNN (Baseline)\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e19.84\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e0.2171\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e63.42\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e8.67\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd class=\"tg-us36\"\u003eOurs\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e21.62\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e0.2514\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e59.61\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e9.49\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n1. To test the network using pretrained models:\n\n\t\tpython3 main.py --arch LstmImg2LastImus --read_features --input_length 5 --sequence_length 10 --output_length 5 --dataset DogClipDataset --features_dir data/new_features --use_test_for_val test --reload /PATH/TO/WEIGHTS/lstm-pred-5-to-5.pytar --image_feature 1024\n\n2. To train the network:\n\n\t\tpython3 main.py --arch LstmImg2LastImus --read_features --input_length 5 --sequence_length 10 --output_length 5 --dataset DogClipDataset --features_dir data/new_features --image_feature 1024\n\n#### Planning like a dog\n\nIn this part we model how dogs plan actions to accomplish a task. To achieve this, we design a task as follows: Given a pair of non-consecutive image frames, plan a sequence of joint movements that the dog would take to get from the first frame (starting state) to the second frame (ending\nstate). To reproduce the results in the paper:\n\n\u003ctable class=\"tg\"\u003e\n\u003ctr\u003e\n\u003cth class=\"tg-us36\"\u003eModel\u003c/th\u003e\n\u003cth class=\"tg-us36\"\u003eTest Accuracy\u003c/th\u003e\n\u003cth class=\"tg-us36\"\u003ePerplexity\u003c/th\u003e\n\u003cth class=\"tg-us36\"\u003eAngular metric\u003c/th\u003e\n\u003cth class=\"tg-us36\"\u003eAll joints\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd class=\"tg-us36\"\u003eCNN(Baseline)\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e14.61\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e0.1419\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e76.18\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e0.14\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd class=\"tg-us36\"\u003eOurs\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e19.77\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e0.2362\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e63.14\u003c/td\u003e\n\u003ctd class=\"tg-us36\"\u003e3.66\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n1. To test the network using pretrained models:\n\n\t\tpython3 main.py --arch LstmImg2ActionPlanning --read_features --planning_distance 5 --sequence_length 7 --output_length 5 --features_dir data/new_features --single_image_feature --image_feature 512 --use_test_for_val test --reload /PATH/TO/WEIGHTS/lstm-plan-5.pytar\n\n2. To train the network:\n\n\t\tpython3 main.py --arch LstmImg2ActionPlanning --read_features --planning_distance 5 --sequence_length 7 --output_length 5 --features_dir data/new_features --single_image_feature --image_feature 512 \n\n\n#### Learning from a dog\n\nTo obtain the representation, we train a ResNet-18 model to estimate the dog movements from time t − 1 to time t by looking at the images at time t − 1 and t. We use a simple Siamese network with two ResNet-18 towers whose weights are shared.\n\nTo test the network:\n\n\tpython3 main.py test --arch FullyConvolutional --output_length 1 --sequence_length 1 --dataset NyuDataset --data data/nyu --no-strict --reload cnn_5_5_best_features.pytar --use_test_for_val --reload /PATH/TO/WEIGHTS/walkable.pytar\n\n\n### Press Coverage\n\n\u003ca href=\"https://techcrunch.com/2018/04/11/whos-a-good-ai-dog-based-data-creates-a-canine-machine-learning-system/\"\u003e\u003cimg src=\"figs/techcrunch.png\" width=\"100px\"\u003e\u003c/a\u003e\n\u0026nbsp;\n\u003ca href=\"https://www.technologyreview.com/s/610775/this-ai-thinks-like-a-dog/\"\u003e\u003cimg src=\"figs/mittechreview.png\" width=\"100px\"\u003e\u003c/a\u003e\n\u0026nbsp;\n\u003ca href=\"http://www.pbs.org/wgbh/nova/next/tech/ai-trained-to-act-like-a-dog/\"\u003e\u003cimg src=\"figs/pbs.png\" width=\"100px\"\u003e\u003c/a\u003e\n\u0026nbsp;\n\u003ca href=\"https://www.theverge.com/2018/4/14/17234570/artificial-intelligence-dogs-research-science-learning\"\u003e\u003cimg src=\"figs/verge.png\" width=\"100px\"\u003e\u003c/a\u003e\n\u0026nbsp;\n\n\u003ca href=\"https://spectrum.ieee.org/tech-talk/robotics/artificial-intelligence/real-dog-behavior-could-inspire-robot-dogs\"\u003e\u003cimg src=\"figs/ieeespectrum.png\" width=\"100px\"\u003e\u003c/a\u003e\n\u0026nbsp;\n\u003ca href=\"https://www.nbcnews.com/mach/science/why-scientists-are-teaching-ai-think-dog-ncna869266\"\u003e\u003cimg src=\"figs/nbc.png\" width=\"50px\"\u003e\u003c/a\u003e\n\u0026nbsp;\n\u003ca href=\"http://www.euronews.com/2018/04/26/why-scientists-are-teaching-ai-think-dog-ncna869266\"\u003e\u003cimg src=\"figs/euronews.png\" width=\"100px\"\u003e\u003c/a\u003e\n\u0026nbsp;\n\u003ca href=\"https://www.reuters.com/video/2018/06/14/dog-vision-project-analyses-canine-behav?videoId=435834651\u0026videoChannel=118065\u0026channelName=Moments+of+Innovation\"\u003e\u003cimg src=\"figs/reuters.png\" width=\"150px\"\u003e\u003c/a\u003e\n\u0026nbsp;\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fehsanik%2FdogTorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fehsanik%2FdogTorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fehsanik%2FdogTorch/lists"}