{"id":16762456,"url":"https://github.com/tlack/dobot-tcn","last_synced_at":"2025-03-16T10:21:21.390Z","repository":{"id":145865551,"uuid":"285734026","full_name":"tlack/dobot-tcn","owner":"tlack","description":"Experiments with Time Contrastive Networks on the Dobot Magician robotic arm","archived":false,"fork":false,"pushed_at":"2020-09-20T02:55:35.000Z","size":56,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-22T22:26:07.087Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tlack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-07T04:08:10.000Z","updated_at":"2020-09-20T02:55:38.000Z","dependencies_parsed_at":"2023-04-06T03:02:32.891Z","dependency_job_id":null,"html_url":"https://github.com/tlack/dobot-tcn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlack%2Fdobot-tcn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlack%2Fdobot-tcn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlack%2Fdobot-tcn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tlack%2Fdobot-tcn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tlack","download_url":"https://codeload.github.com/tlack/dobot-tcn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243853626,"owners_count":20358452,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T04:44:47.398Z","updated_at":"2025-03-16T10:21:21.361Z","avatar_url":"https://github.com/tlack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dobot-tcn\n\nThis is an attempt to utilize a time contrastive network to control a robotic arm.\n\nIt supports the Dobot Magician and custom Makeblock Auriga / MS-12a configurations.\n\nThis experiment is an attempt to replicate\n[Time Contrastive Networks: Self-Supervised Learning from Multiview Observation](https://sermanet.github.io/tcn/).\n\n\n## status\n\nDATA: \nGeneralized data recording subsystem works. There are two different variants, one of which records a video, the other just still frames.\nUse the stills, bro. See [my other repo](https://github.com/tlack/dobot-tcn-training-data/) for samples of training data.\n\nTRAINING: \nThere are two failed direct image regression (picture of robot -\u003e joint positions of robot) attempts here. Others may\nhave luck with that approach. \n\nI seem to be having better luck with a contrastive model, a hacky manual version of which can be seen here:\n[on Google Colab](https://colab.research.google.com/drive/18axcd2EtWSp9H5PnxqJe6arKzTMDBgNC?usp=sharing) or [YT](https://www.youtube.com/watch?v=f2J2HG72fd8). Needs a lot of work.\n\nIMITATION:\nThe last step is to use this trained model and a system like [PPO](https://openai.com/blog/openai-baselines-ppo/) to \nactually control a robot with a given video, or, the horror, a live stream. \n\nThis part hasn't been started.\n\n## prereq\n\n1. A robotic arm. We support: \n\t- Dobot Magician\n   \t- Makeblock Auriga with MS-12a Smart Sensors\n\t- Dynamixel XL (soon!)\n2. PiCamera x2 or more\n3. Python3.7+\n4. ffmpeg\n\n## what's here\n\n### picamera_webserver.py\n\nThis is a simple Flask web app to interact with a Raspberry Pi PiCamera. \n\nEdit settings on first line and run.\n\nStores videos in `data/picamera-webserver`.\n\nThe API presents these endpoints:\n\n`http://0.0.0.0:5858/` - **index**\nReturns listing of known videos\n\n`http://0.0.0.0:5858/start?fname=test.h264` - **start**\nStart recording to named video file\n\n`http://0.0.0.0:5858/start?fname=test.h264` - **stop**\nStop recording - file name optional.\n\n`http://0.0.0.0:5858/send?fname=test.h264` - **send video**\nRetrieve contents of video. Returned as Base64 string inside JSON structure:\n```{\"fn\": \"test.h264\", \"data\": \"...enormous string...\", \"size\": 123123}```\n\nRun one copy of this script on each PiCamera node.\n\n### pydobot/\n\nThis is my fork of the regular Pydobot code.\n\n### auriga/\n\nSupport for Makeblock Auriga motor controller board with ATmel Mega2650, along with\ntheir MS-12a Smart Servos.\n\n#### auriga/Makeblock-Auriga-SmartServoMS12a-FIRMWARE\n\nThis is an Arduino sketch that provides new firmware for your Auriga. This new firmware\nis a highly optimized ASCII protocol that allows you to control your bot over serial\nfrom your Raspberry Pi. \n\nNote that Makeblock provides some connectivity with their standard firmware but the\nprotocol isn't well documented.\n\n**Load this on your Auriga first** before continuing to the Python stage.\n\n#### auriga/Auriga.py\n\nThis is a PySerial-based wrapper that speaks to the firmware and provides a\nsuper simple class wrapper.\n\n### jointrecording.py\n\nThis is the main script that enables us to begin data collection on the Dobot Magician.\n\nOpen the file. You will notice a list of camera nodes at the top. Edit to match your cameras and run.\n\nData is stored under `data/jointrecording/`. It names each one after an experiment name you enter.\n\nHere is how it works:\n\n* Initialize robot.\n\n* Prompt for experiment name (enter for same name as last time)\n\n* Create folder `data/jointrecording/[experiment_name]-[timestamp]`. This is called the data dir.\n\n* For N_EXAMPLES as N..\n\n* * Record current robot position including joints - this is called a pose.\n\n* * Instruct all cameras to begin recording.\n\n* * Select new location to move to - this is called a goal. Only one axis will be tweaked at a time. The number of repetitions per axis is in AXIS_REPS. The amount of change in each value is in SWEEP_RANGE.\n\n* * Move to goal.\n\n* * Record new pose.\n\n* * Stop all cameras\n\n* * Log to pose to `[datadir]/[N]-[timestamp]-joints.json`\n\n* * Receive videos from all cameras and store each to `[datadir]/[N]-[timestamp]-[camera_name].h264`\n\n* * Return to original pose\n\nThis should be useful for training, but I'm not there yet.\n\n### 2upvideo.sh\n\nThis takes two matching videos and joins them into a single side-by-side video.\n\nIf called with a folder, it will perform this operation for all experiment videos in the folder.\n\nIn folder mode, It will attempt to create a single montage called ALL.h264 at the end. This sometimes fails.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftlack%2Fdobot-tcn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftlack%2Fdobot-tcn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftlack%2Fdobot-tcn/lists"}