{"id":21739734,"url":"https://github.com/berkeleyautomation/dvrk-vismpc","last_synced_at":"2026-05-09T15:41:54.242Z","repository":{"id":90033442,"uuid":"233959917","full_name":"BerkeleyAutomation/dvrk-vismpc","owner":"BerkeleyAutomation","description":"DVRK code for VisMPC + Fabric project","archived":false,"fork":false,"pushed_at":"2020-05-30T18:35:53.000Z","size":474,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-25T21:26:41.200Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BerkeleyAutomation.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-14T23:59:29.000Z","updated_at":"2023-06-28T07:12:34.000Z","dependencies_parsed_at":"2023-05-19T08:45:25.790Z","dependency_job_id":null,"html_url":"https://github.com/BerkeleyAutomation/dvrk-vismpc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerkeleyAutomation%2Fdvrk-vismpc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerkeleyAutomation%2Fdvrk-vismpc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerkeleyAutomation%2Fdvrk-vismpc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BerkeleyAutomation%2Fdvrk-vismpc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BerkeleyAutomation","download_url":"https://codeload.github.com/BerkeleyAutomation/dvrk-vismpc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244717336,"owners_count":20498283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-26T06:10:23.621Z","updated_at":"2026-05-09T15:41:49.223Z","avatar_url":"https://github.com/BerkeleyAutomation.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Da Vinci Research Kit (dVRK) Code for Fabrics and Visual MPC\n\n*Update May 2020*: this is the code we used for the physical fabrics\nexperiments with the dVRK. The master branch has the code for\nour RSS 2020 paper \"VisuoSpatial Foresight (VSF) for Multi-Step, Multi-Task\nFabric Manipulation\":\n\n```\n@inproceedings{fabric_vsf_2020,\n    author = {Ryan Hoque and Daniel Seita and Ashwin Balakrishna and Aditya Ganapathi and Ajay Tanwani and Nawid Jamali and Katsu Yamane and Soshi Iba and Ken Goldberg},\n    title = {{VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation}},\n    booktitle = {Robotics: Science and Systems (RSS)},\n    Year = {2020}\n}\n```\n\nThe paper is [available on arXiv][2] and we have additional information on the\n[project website][3], along with [code for the simulator and training video\nprediction models][4].\n\n*Disclaimer*: some parts of the code might not be documented to the best of our\nability. If you have questions on how to use the code, please contact Daniel\nSeita (`seita@berkeley.edu`) with details on what you want to do, and I shall\ndo my best to help you out.\n\n\u003chr\u003e\n\n# Original README\n\nThis is not for the original smoothing paper. For that, [see this code][1].\n\nUse the following core API:\n\n- `dvrkArm.py`: an API wrapping around the ROS messages to get or set positions.\n- `dvrkClothSim.py` for moving the arms.\n\n\n## Installation and Setup\n\nWe're using davinci-arm machine (formerly named davinci0).\n\nWe use a mix of the system Python 2.7 on there, and a Python 3.5 virtual env.\nUnlike in the original smoothing paper (from fall 2019) the camera code has\nbeen updated to use Python 3. However, the neural network code (as before)\nstill requires Python 3.\n\nTo install, first clone this repository. Then make a new Python 3.5 virtualenv\nfrom a reference virtualenv on davinci0:\n\n```\nsource ~/venv/bin/activate\n```\n\nI did `pip freeze` and put this in a file, and then made a new Python3 virtualenv:\n\n```\nvirtualenv venvs/py3-vismpc-dvrk  --python=python3\n```\n\nand then did `pip install -r [filename].txt` to get all the packages updated.\nAfter that, running `import zivid` should work. This should be the same\nvirtualenv that we use for loading TensorFlow.\n\n**Update: not sure what happened but we may have to install a Python2\nvirtualenv as well that uses the system site packages, so that we can do `pip\ninstall scikit-learn` and get the structural similarity code ... stay tuned!**\n\n## Calibration\n\nBefore using the robot, we need to calibrate it to ensure that we have a\nmapping from coordinates in `[-1,1]` to coordinates w.r.t. the robot's base\nframe. Put a 5x5 checkerboard on top of the foam rubber. Then go into the\n`tests/` folder and run `python test_03_checkerboard_pick_test.py` to check the\ncalibration. Use the system Python (it's Python 2.7). Once it looks good, save\nthe calibration file (!!) and then move on to experiments.\n\n## Experimental Usage\n\nPerforming our experiments involves several steps, due to loading a separate\nneural network and having it run continuously. Roughly, the steps are:\n\n0. Check the configuration file in `config.py` which will contain a bunch of\nsettings we can adjust. *In particular, adjust which calibration data we should\nbe using*. They are in the form of text files. There is a SECOND config file in\nthe network loading folder.\n\n1. Activate the robot via `roscore`, then (in a separate tab) run `./teleop` in\n`~/catkin_ws` and click HOME. This gets the dvrk setup.\n\n2. In another tab, *activate the Python 3 virtualenv above*, and run\n\n   ```\n   rm dir_for_imgs/*.png ; rm dir_for_imgs/result*.txt ; python call_network/load_net.py\n   ```\n   See `call_network/README.md` for detailed instructions.  This script runs\n   continuously in the background and checks for any new images in the target\n   directory. Removes images in the calibration directory.\n\n3. In another tab, *activate the Python 3 virtualenv above*, and run `python\nZividCapture.py`. This script runs continuously and will activate with a\nkeyboard command. Whenever we need a new set of images, we need to press enter\nat this TAB. Then that will take a picture, and save images indexed by a\nleading number. The neural net code will detect that number and load a specific\nfile w/that index.\n\n4. Finally, run `python run.py --tier X` for experiments, using the system\nPython. This requires some keyboard strokes. *The code runs one episode* and\nthen terminates.  Repeat for more episodes.\n\n## Tips\n\n- To test the camera code, just run `python ZividCapture.py` and adjust the\n  main method. Must be in the Python3 virtualenv. The run script will do some\n  similar image processing, but it should not be an issue if it's in Python2, I\n  don't think image processing code changed that much.\n\n- For quick testing of the da vinci with resorting to the machinery of the\n  experimental pipeline, use `python dvrkClothSim.py` (for generic motion) or\n  one of the test scripts in `tests/` (for calibration accuracy).\n\n\n## Quick Start\n\nHere's a minimal working example, e.g., could be in `tests/test_01_positions.py`:\n\n```python\nimport sys\nsys.path.append('.')\nimport dvrkArm\n\nif __name__ == \"__main__\":\n    p = dvrkArm.dvrkArm('/PSM1')\n    pose_frame = p.get_current_pose_frame()\n    pose_rad = p.get_current_pose()\n    pose_deg = p.get_current_pose(unit='deg')\n\n    print('pose_frame:\\n{}'.format(pose_frame))\n    print('pose_rad: {}'.format(pose_rad))\n    print('pose_deg: {}'.format(pose_deg))\n\n    targ_pos = [0.04845971,  0.05464517, -0.12231114]\n    targ_rot = [4.65831872, -0.69974499,  0.87412989]\n    p.set_pose(pos=targ_pos, rot=targ_rot, unit='rad')\n```\n\nand run `python tests/test_01_positions.py`. This will print out pose\ninformation about the dvrk arm, which can then be used to hard code some\nactions. For example, above we show how to set the pose of the arm given\npre-computed `targ_pos` and `targ_rot`.\n\n\n\n## Using DGX and Docker for SV2P\n\nThis will be a bit hacky but we can do it. Here's what I do *for each episode*.\n\nMake sure I start up the camera script:\n\n```\nrm dir_for_imgs/*.png ; rm dir_for_imgs/*.txt ; python ZividCapture.py\n```\n\nand the run script.\n\n```\npython run.py --tier 1 --vf\n```\n\nDo NOT run the network loading script.\n\nOn the DGX:\n\n```\nnvidia-docker run --runtime=nvidia -it -e NVIDIA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 --rm -v /raid/for-daniel/:/data saved-visual-cuda10-ashwin bash\n```\n\nThis will start something Docker-related based on Ashwin's CUDA 10 \"image\"\n(i.e., \"recipe\") where containers are *instances* of an image.\n\nInside that docker container, activate the virtualenv:\n\n```\nsource /data/envs/visual/bin/activate\n```\n\nRun something like this (assuming GPU 5 is available, check with `nvidia-smi`):\n\n```\n(visual) root@23db4c61657c:/data/cloth-visual-mpc# rm /data/dir_for_imgs/* ; CUDA_VISIBLE_DEVICES=5 python vismpc/scripts/dvrk.py\n```\n\n**IF MAKING CHANGES TO THE VISMPC CODE BE SURE TO COMPILE VIA `python setup.py\ninstall`, EVEN IF WE ARE JUST CHANGING C.E.M. HYPERPARAMETERS.**\n\nIt takes a few seconds to start up. Eventually, this will run continuously like\nmy other script that works locally here (`call_network/load_net.py`).\n\nOn the local surgical robot machine, make sure we keep scp'ing BOTH the color\nand depth images. In addition I have to ensure that I am scp-ing the *latest\nversion* of them, in numerical order, and that I an scp-ing the *latest action*\ncorrectly. This means a cycle that starts off with these commands. First, after\ngetting the camera images, do this:\n\n```\nscp dir_for_imgs/000-* seita@jensen.ist.berkeley.edu:/raid/for-daniel/dir_for_imgs/\n```\n\nthen on the DGX, it will load SV2P and run MPC. It will save an action, indexed\nat one number next. Get this back from the DGX:\n\n```\nscp seita@jensen.ist.berkeley.edu:/raid/for-daniel/dir_for_imgs/*_001.txt dir_for_imgs/\n```\n\n(and putting it in the same directory as images).  Then let the robot run, and\nonce it's done, get the next camera images, and scp with the next index:\n\n```\nscp dir_for_imgs/001-* seita@jensen.ist.berkeley.edu:/raid/for-daniel/dir_for_imgs/\n```\n\nthen after MPC:\n\n```\nscp seita@jensen.ist.berkeley.edu:/raid/for-daniel/dir_for_imgs/*_002.txt dir_for_imgs/\n```\n\nand so on! I modified Ryan's script so that it looks at `/data/dir_for_images/`\nwhich is where `/raid/for-daniel` goes. Well, actually to make things simple,\nyou can make the commands the same both ways:\n\n\n```\nscp dir_for_imgs/*56.png seita@jensen.ist.berkeley.edu:/raid/for-daniel/dir_for_imgs/\n```\n\nand\n\n```\nscp seita@jensen.ist.berkeley.edu:/raid/for-daniel/dir_for_imgs/*.txt dir_for_imgs/\n```\n\nThe scripts we use should only select for the most recent samples.\n\n[1]:https://github.com/BerkeleyAutomation/dvrk_python\n[2]:https://arxiv.org/abs/2003.09044\n[3]:https://sites.google.com/view/fabric-vsf\n[4]:https://github.com/ryanhoque/fabric-vsf\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fberkeleyautomation%2Fdvrk-vismpc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fberkeleyautomation%2Fdvrk-vismpc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fberkeleyautomation%2Fdvrk-vismpc/lists"}