{"id":28516105,"url":"https://github.com/labelbox/lbx-franka-teach","last_synced_at":"2025-07-05T17:30:42.460Z","repository":{"id":295325128,"uuid":"989759490","full_name":"Labelbox/lbx-Franka-Teach","owner":"Labelbox","description":"OpenTeach fork for the bimanual Franka Research 3 setup","archived":false,"fork":false,"pushed_at":"2025-06-02T09:54:17.000Z","size":1801,"stargazers_count":0,"open_issues_count":6,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-09T03:39:49.667Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Labelbox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-24T19:10:44.000Z","updated_at":"2025-05-31T04:06:46.000Z","dependencies_parsed_at":"2025-06-01T00:28:01.065Z","dependency_job_id":null,"html_url":"https://github.com/Labelbox/lbx-Franka-Teach","commit_stats":null,"previous_names":["labelbox/lbx-franka-teach"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Labelbox/lbx-Franka-Teach","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Labelbox%2Flbx-Franka-Teach","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Labelbox%2Flbx-Franka-Teach/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Labelbox%2Flbx-Franka-Teach/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Labelbox%2Flbx-Franka-Teach/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Labelbox","download_url":"https://codeload.github.com/Labelbox/lbx-Franka-Teach/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Labelbox%2Flbx-Franka-Teach/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263778255,"owners_count":23509962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-09T03:32:14.653Z","updated_at":"2025-07-05T17:30:42.454Z","avatar_url":"https://github.com/Labelbox.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Franka-Teach\n\nA comprehensive teleoperation and data collection system for Franka robots, featuring VR control via Meta Quest, MCAP data recording, and simulation support.\n\n## Features\n\n- **VR Teleoperation**: Full 6DOF control using Meta Quest/Oculus controllers\n- **MCAP Recording**: DROID-compatible data format for robot learning\n- **Multi-Camera Support**: Intel RealSense and ZED camera integration\n- **Simulation Mode**: Test without hardware using FR3 robot simulation\n- **Real-time Control**: 15-30Hz control loop with safety features\n- **High-Performance Async Architecture**: 40Hz data recording with 6.6Hz robot control\n- **Hot Reload**: Automatic server restart on code changes\n- **Performance Mode**: Optimized for high-frequency control and recording\n- **Data Verification**: Automatic validation of recorded trajectories\n\n## Table of Contents\n\n- [System Requirements](#system-requirements)\n- [Installation](#installation)\n  - [NUC Setup](#nuc-setup)\n  - [Lambda Machine Setup](#lambda-machine-setup)\n- [Quick Start](#quick-start)\n- [VR Teleoperation](#vr-teleoperation)\n- [Data Recording](#data-recording)\n- [Camera Setup](#camera-setup)\n- [Simulation Mode](#simulation-mode)\n- [Troubleshooting](#troubleshooting)\n- [Additional Documentation](#additional-documentation)\n\n## System Requirements\n\n### Hardware\n- Franka Emika Robot (FR3 or Panda)\n- NUC computer with real-time kernel\n- Lambda workstation or similar\n- Meta Quest 2/3 or Oculus headset\n- Intel RealSense or ZED cameras (optional)\n\n### Software\n- Ubuntu 22.04 (NUC) / Ubuntu 20.04+ (Lambda)\n- CUDA-capable GPU (for Lambda machine)\n- Python 3.10+\n- Real-time kernel (NUC only)\n\n## Installation\n\n### NUC Setup\n\n1. **Install Ubuntu 22.04 with real-time kernel**\n   ```bash\n   # Follow instructions at:\n   # https://frankaemika.github.io/docs/installation_linux.html#setting-up-the-real-time-kernel\n   ```\n\n2. **Configure network settings**\n   - Set static IP for robot network interface\n   - Ensure NUC can communicate with robot at 172.16.0.4 (right) and 172.16.1.4 (left)\n\n### Lambda Machine Setup\n\n1. **Clone and setup deoxys_control**\n   ```bash\n   git clone git@github.com:NYU-robot-learning/deoxys_control.git\n   cd deoxys_control/deoxys\n   \n   # Create conda environment\n   mamba create -n \"franka_teach\" python=3.10\n   conda activate franka_teach\n   \n   # Install deoxys (select version 0.13.3 for libfranka when prompted)\n   ./InstallPackage\n   make -j build_deoxys=1\n   pip install -U -r requirements.txt\n   ```\n\n2. **Clone and setup Franka-Teach**\n   ```bash\n   git clone \u003cyour-franka-teach-repo\u003e\n   cd Franka-Teach\n   pip install -r requirements.txt\n   ```\n\n3. **Install ReSkin sensor library (optional)**\n   ```bash\n   git clone git@github.com:NYU-robot-learning/reskin_sensor.git\n   cd reskin_sensor\n   pip install -e .\n   ```\n\n4. **Setup Oculus Reader for VR control**\n   ```bash\n   cd oculus_reader_app\n   # Follow instructions in oculus_reader_app/README.md\n   ```\n\n5. **Configure IP Addresses** ⚠️ **Important**\n   \n   You must update the IP addresses in the configuration files to match your network setup:\n   \n   ```bash\n   # Edit the deoxys configuration files\n   cd frankateach/configs\n   \n   # For right robot:\n   vim deoxys_right.yml\n   # Update:\n   # - PC.IP: Your Lambda machine's IP address\n   # - NUC.IP: Your NUC's IP address (same as PC.IP if running on same machine)\n   # - ROBOT.IP: Your Franka robot's IP address (e.g., 192.168.1.59)\n   \n   # For left robot (if applicable):\n   vim deoxys_left.yml\n   # Update the same fields with appropriate IP addresses\n   ```\n   \n   **Note**: The `franka_server.py` will fail to connect with \"Waiting for the robot to connect...\" \n   if these IP addresses don't match your actual network configuration.\n\n### Network Proxy Setup\n\n1. **Install FoxyProxy extension** in Chrome/Firefox\n\n2. **Configure SSH host** in `~/.ssh/config`:\n   ```\n   Host nuc\n       HostName 10.19.248.70\n       User robot-lab\n       LogLevel ERROR\n       DynamicForward 1337\n   ```\n\n3. **Configure FoxyProxy** to use SOCKS5 proxy on localhost:1337\n\n## Quick Start\n\n### 1. Connect to Robot\n\n```bash\n# SSH into NUC\nssh nuc\n\n# Access Franka Desk interface via browser (with proxy enabled)\n# Right robot: http://172.16.0.4/desk\n# Left robot: http://172.16.1.4/desk\n# Username: GRAIL\n# Password: grail1234\n```\n\n### 2. Enable Robot\n\nIn Franka Desk interface:\n1. Click \"Unlock joints\" (open brakes)\n2. Enable FCI mode\n3. If gripper issues: Settings → End Effector → Power cycle → Re-initialize\n\n### 3. Start Deoxys Control (on NUC)\n\n```bash\n# Terminal 1: Start arm control\ncd /home/robot-lab/work/deoxys_control/deoxys\n./auto_scripts/auto_arm.sh config/franka_right.yml  # or franka_left.yml\n\n# Terminal 2: Start gripper control (optional)\n./auto_scripts/auto_gripper.sh config/franka_right.yml\n```\n\n### 4. Start Servers (on Lambda)\n\n```bash\n# Terminal 1: Start Franka server\ncd /path/to/Franka-Teach\npython franka_server.py\n\n# Terminal 2: Start camera server (optional)\npython camera_server.py --config camera_config_example.json\n```\n\n### 5. Start VR Teleoperation with Recording\n\n```bash\n# Terminal 3: Start Oculus VR server with MCAP recording\npython oculus_vr_server.py\n\n# Or with specific options:\npython oculus_vr_server.py --left-controller  # Use left controller\npython oculus_vr_server.py --simulation       # Test in simulation\npython oculus_vr_server.py --debug           # Debug mode (no robot control)\npython oculus_vr_server.py --no-recording    # Disable MCAP recording\n```\n\n## VR Teleoperation\n\n### Controller Setup\n\n1. **Connect Quest headset** via USB or WiFi\n2. **Test connection**: `python test_oculus_reader.py`\n3. **Calibrate forward direction**: Hold joystick + move controller forward\n\n### Control Scheme\n\n#### Right Controller (default)\n- **Hold Grip**: Enable robot movement\n- **Release Grip**: Pause movement (robot stays in place)\n- **Hold Trigger**: Close gripper\n- **Release Trigger**: Open gripper\n- **Joystick Press**: Reset controller orientation\n\n#### Recording Controls (when enabled)\n- **A Button**: Start new recording / Reset current recording\n- **B Button**: Mark recording as successful and save\n\nSee [oculus_control_readme.md](oculus_control_readme.md) for detailed VR control instructions.\n\n## Data Recording\n\nThe system supports MCAP data recording in Labelbox Robotics format:\n- Press **A button** to start/reset recording\n- Press **B button** to mark recording as successful and save\n- Recordings are saved to `~/recordings/success/` or `~/recordings/failure/`\n- See [MCAP_RECORDING_README.md](MCAP_RECORDING_README.md) for details\n\n### Visualizing Robot in Foxglove Studio\n\nThe MCAP recordings include the robot URDF model and joint states for visualization:\n- See [FOXGLOVE_ROBOT_VISUALIZATION.md](FOXGLOVE_ROBOT_VISUALIZATION.md) for setup instructions\n- Test with `python3 test_foxglove_robot.py` to create a sample file\n\n## Camera Setup\n\n### Configure Cameras\n\nEdit `\n\n## Simulation Mode\n\n### Simulation Setup\n\n1. **Test in simulation**: `python oculus_vr_server.py --simulation`\n2. **Test without hardware**: Use FR3 robot simulation\n\n## Troubleshooting\n\n### Low Recording Frequency\n- Ensure performance mode is enabled (default with `run_server.sh`)\n- Check CPU usage and reduce other processes\n- Verify in console output: should show ~39-40Hz\n\n### Robot Communication Slow\n- The ~149ms latency is hardware limited\n- System automatically uses predictive control\n- Check network/USB connection quality\n\n### VR Controller Not Detected\n- Ensure Oculus is connected via USB or network\n- Check with `adb devices` for USB connection\n- For network: use `--ip \u003cquest-ip-address\u003e`\n\n### Franka Server Connection Issues\nIf `franka_server.py` shows \"Waiting for the robot to connect...\" indefinitely:\n- **Check IP configuration**: The most common cause is incorrect IP addresses in `frankateach/configs/deoxys_right.yml` or `deoxys_left.yml`\n- **Verify network connectivity**: Can you ping the robot from the Lambda machine?\n- **Check if deoxys is already running**: Only one process can connect to the robot at a time\n  ```bash\n  # Check for running deoxys processes\n  ps aux | grep -E \"(franka-interface|gripper-interface)\"\n  # Kill if necessary\n  sudo pkill -f \"franka-interface|gripper-interface\"\n  ```\n- **Ensure robot is in FCI mode**: Check Franka Desk interface\n\n## Additional Documentation\n\n### Architecture Overview\n\nThe system uses a sophisticated asynchronous architecture to achieve high-frequency data recording (40Hz) while managing slower robot communication (6.6Hz):\n\n```\nVR Thread (50Hz) → Control Thread (40Hz) → Recording Thread (40Hz)\n                          ↓\n                   Robot Comm Thread → Robot Hardware (6.6Hz)\n```\n\n**Key Benefits:**\n- 6x higher recording rate than traditional synchronous approaches\n- Non-blocking operation ensures smooth teleoperation\n- Predictive control maintains responsiveness\n- Thread-safe data handling with minimal lock contention\n\nFor detailed architecture documentation, see [ASYNC_ARCHITECTURE_README.md](ASYNC_ARCHITECTURE_README.md).\n\n### Performance Optimization\n\nThe system includes several performance optimizations:\n\n#### Performance Mode (Default)\n- Control frequency: 40Hz (2x base rate)\n- Position gain: 10.0 (100% higher)\n- Rotation gain: 3.0 (50% higher)\n- Optimized for tight tracking\n\n#### Async Features\n- **Decoupled threads** for VR, control, recording, and robot I/O\n- **Predictive control** when robot feedback is delayed\n- **Non-blocking queues** for data flow\n- **Lock-free design** where possible\n\nSee [ASYNC_ARCHITECTURE_README.md](ASYNC_ARCHITECTURE_README.md) for detailed performance documentation.\n\n### Hot Reload\n\nEnable automatic server restart on code changes:\n\n```bash\n./run_server.sh --hot-reload\n```\n\nThis monitors Python files and configs, restarting the server when changes are detected. Perfect for rapid development and testing.\n\nSee [HOT_RELOAD_README.md](HOT_RELOAD_README.md) for details.\n\n### Data Format\n\nRecordings are saved in MCAP format with the following structure:\n\n- `/robot_state`: Joint positions, cartesian pose, gripper state (40Hz)\n- `/action`: Velocity commands sent to robot (40Hz)\n- `/vr_controller`: Raw VR controller data (40Hz)\n- `/tf`: Transform tree for visualization\n- `/joint_states`: ROS-compatible joint states\n\n### Documentation\n\n- [Async Architecture \u0026 Performance](ASYNC_ARCHITECTURE_README.md)\n- [Hot Reload Feature](HOT_RELOAD_README.md)\n- [MCAP Recording Format](MCAP_RECORDING_README.md)\n- [Foxglove Visualization](FOXGLOVE_ROBOT_VISUALIZATION.md)\n- [VR Robot Mapping](VR_ROBOT_MAPPING.md)\n\n### Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make changes and test with hot reload\n4. Submit a pull request\n\n### License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n### Acknowledgments\n\n- Based on the DROID VRPolicy implementation\n- Uses Deoxys control framework for Franka robots\n- MCAP format for efficient data storage","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flabelbox%2Flbx-franka-teach","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flabelbox%2Flbx-franka-teach","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flabelbox%2Flbx-franka-teach/lists"}