{"id":17686984,"url":"https://github.com/mttgdd/robotcardataset-scraper","last_synced_at":"2025-05-07T14:23:09.519Z","repository":{"id":187104167,"uuid":"175678357","full_name":"mttgdd/RobotCarDataset-Scraper","owner":"mttgdd","description":"Downloads files matching patterns from the Oxford Robotcar Dataset website","archived":false,"fork":false,"pushed_at":"2022-09-15T12:45:14.000Z","size":48,"stargazers_count":50,"open_issues_count":4,"forks_count":24,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-07T14:22:53.292Z","etag":null,"topics":["datasets","gps","laser","learning","lidar","monocular","python","robotics","stereo","vision","website"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mttgdd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-03-14T18:31:07.000Z","updated_at":"2025-04-18T13:42:24.000Z","dependencies_parsed_at":"2023-08-09T01:26:07.475Z","dependency_job_id":null,"html_url":"https://github.com/mttgdd/RobotCarDataset-Scraper","commit_stats":null,"previous_names":["mttgdd/robotcardataset-scraper"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mttgdd%2FRobotCarDataset-Scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mttgdd%2FRobotCarDataset-Scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mttgdd%2FRobotCarDataset-Scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mttgdd%2FRobotCarDataset-Scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mttgdd","download_url":"https://codeload.github.com/mttgdd/RobotCarDataset-Scraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252892752,"owners_count":21820697,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datasets","gps","laser","learning","lidar","monocular","python","robotics","stereo","vision","website"],"created_at":"2024-10-24T10:46:36.866Z","updated_at":"2025-05-07T14:23:09.498Z","avatar_url":"https://github.com/mttgdd.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nThis script can be used to perform sequenced downloads of various sensors logs for the robotcar-dataset \u003chttps://robotcar-dataset.robots.ox.ac.uk/\u003e.\n\nThe code is tested for Python3 on Ubuntu 16.04.\n\n# Getting started\n\nA prebuilt docker image is available with:\n\n```bash\ndocker pull matthewgadd/robotcar-dataset-scraper:latest\n```\n\nOtherwise, you can build the docker image from scratch:\n\n```bash\nwget https://raw.githubusercontent.com/matthewgadd/RobotCarDataset-Scraper/master/Dockerfile\ndocker build -t matthewgadd/robotcar-dataset-scraper:latest .\n```\n\n# Example usage\n\nMount your downloads dir through the docker image:\n\n```bash\ndocker run --rm -it -w /RobotCarDataset-Scraper -v $HOME/Downloads:/Downloads matthewgadd/robotcar-dataset-scraper:latest\n```\n\nThe script\n\n```bash\npython get_datasets.py\n```\n\nparses the html for the complete dataset listing and scrapes each dataset page and retrieves the available files (tar) for download. With the example output file datasets.csv provided, you should then run:\n\n```bash\npython scrape_mrgdatashare.py --downloads_dir /Downloads --datasets_file datasets.csv --username USERNAME --password PASSWORD\n```\n\nIf you do not need the full dataset, you can use `--choice_sensors` and `--choice_runs_file` options to download parts of the dataset on your own needs. \n\n* `--choice_sensors` option can receive multiple sensor names in `tags, stereo_centre, stereo_left, stereo_right, vo, mono_left, mono_right, mono_rear, lms_front, lms_rear, ldmrs, gps, all. `\n* `--choice_runs_file` option receive a `.txt` file that contains the names of runs you want to download, we provide an example filea sample file `example_list.txt` .\n\nfor example you can download \"stereo_centre\", \"vo\" and \"lms_front\" data of  \"2014-05-19-13-20-57\" and \"2014-06-26-09-31-18\" by the following command:\n\n```bash\npython scrape_mrgdatashare.py --choice_sensors stereo_centre,vo,lms_front --choice_runs_file example_list.txt --downloads_dir /Downloads --datasets_file datasets.csv --username USERNAME --password PASSWORD\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmttgdd%2Frobotcardataset-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmttgdd%2Frobotcardataset-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmttgdd%2Frobotcardataset-scraper/lists"}