{"id":30115519,"url":"https://github.com/stonet2000/simple-easyhec","last_synced_at":"2025-08-10T08:20:00.712Z","repository":{"id":308453899,"uuid":"1032707268","full_name":"StoneT2000/simple-easyhec","owner":"StoneT2000","description":"Implementation of EasyHec for automatic accurate camera calibration","archived":false,"fork":false,"pushed_at":"2025-08-06T01:04:32.000Z","size":1103,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-06T03:06:41.883Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StoneT2000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-3RD-PARTY","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-05T17:45:04.000Z","updated_at":"2025-08-06T02:21:08.000Z","dependencies_parsed_at":"2025-08-06T03:06:43.770Z","dependency_job_id":"87e2139c-8efe-46f2-866b-b895e7e70e55","html_url":"https://github.com/StoneT2000/simple-easyhec","commit_stats":null,"previous_names":["stonet2000/simple-easyhec"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/StoneT2000/simple-easyhec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StoneT2000%2Fsimple-easyhec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StoneT2000%2Fsimple-easyhec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StoneT2000%2Fsimple-easyhec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StoneT2000%2Fsimple-easyhec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StoneT2000","download_url":"https://codeload.github.com/StoneT2000/simple-easyhec/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StoneT2000%2Fsimple-easyhec/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269693614,"owners_count":24460256,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-10T02:00:08.965Z","response_time":71,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-10T08:19:56.292Z","updated_at":"2025-08-10T08:20:00.695Z","avatar_url":"https://github.com/StoneT2000.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple EasyHec: Accurate and Automatic Camera Calibration\n\nThis repo provides (mostly) pip installable code to easily calibrate a camera (mounted and not mounted) and get its extrinsics with respect to some object (like a robot). It is a cleaned up and simplified version of the original [EasyHeC project](https://github.com/ootts/EasyHeC). It works by taking a dataset of segmentation masks of the object and the object poses, and based on the camera intrinsics and an initial guess of the extrinsics, optimizes that guess into an accurate estimate of the true extrinsics. The optimization process leverages [Nvdiffrast](https://github.com/NVlabs/nvdiffrast) for differentiable rendering to run this optimization process.\n\nImage below shows three different segmentation overlays on top of an RGB image in simulation generated by rendering segmentation from various camera poses of a 3rd-view camera. It compares the initial bad extrinsic guess, the optimized estimate, and the ground truth.\n\n![](./assets/sim_example.png)\n\nAnother example below shows it working for a mounted camera. Notice that the initial extrinsic guess is so bad that the segmentation covers the whole image.\n\n![](./assets/sim_example_mounted_camera.png)\n\n## Installation\n\nInstall from pip\n```bash\npip install easyhec\npip install \"nvdiffrast @ git+https://github.com/NVlabs/nvdiffrast.git@729261dc64c4241ea36efda84fbf532cc8b425b8\"\n```\n\nInstall from git (We highly recommend you create a conda/mamba environment to manage packages)\n\n```bash\ngit clone https://github.com/StoneT2000/simple-easyhec\nconda create -n easyhec \"python==3.11\"\npip install -e .\npip install \"nvdiffrast @ git+https://github.com/NVlabs/nvdiffrast.git@729261dc64c4241ea36efda84fbf532cc8b425b8\"\n```\n\nThe code relies on Nvdiffrast which can sometimes be tricky to setup as it can have some dependencies that need to be installed outside of pip. If you have issues installing Nvdiffrast (which provides dockerized options) see their [documentation](https://nvlabs.github.io/nvdiffrast/) or use our google colab script.\n\nFor those who don't want to manually segment their robot images you can use [SAM2](https://github.com/facebookresearch/sam2). Follow the installation instructions in that repo to set it up locally. Otherwise this repo provides a simple point annotation interface to annotate provided images with SAM2 to generate segmentation maps.\n\n## Usage\n\n### Simulation\n\nTo test in simulation we provide an example through maniskill. The example generates 5 training synthetic images of the Franka/Panda robot with segmentation masks of the robot in random joint positions sampled around the initial joint position, and an initial extrinsic camera guess noisly sampled around the ground truth. By default 10 test images of the robot in different configurations are tested on and saved locally to the `results/` folder. This simulation also has raytracing for nicer renders which is disabled by default, turn it on by adding --shader rt\n\n```bash\npip install easyhec[sim-maniskill]\n# Franka arm, camera used here is a offhand 512x512 camera\npython -m easyhec.examples.sim.maniskill -e StackCube-v1 --samples 5\n```\n\nYou can also try out other robots and cameras that ManiSkill provides in other environments like the base camera for SO100\n\n\n```bash\n# SO100 arm, base_camera here is a 128x128 camera \npython -m easyhec.examples.sim.maniskill -e SO100GraspCube-v1 --samples 5 \\\n    --camera-name base_camera \\\n    --initial-extrinsic-guess-rot-error 15 --initial-extrinsic-guess-pos-error 0.1\n```\n\nWrist cameras are also possible but are harder to get working. The amount of initial extrinsic error must also be lower since the robot already takes up a large part of the image. If the robot is far away more error is possible to solve from. This is why a specific seed is specified for the script below (the seed determines the direction of error we generate for testing, in some directions its impossible because of occlusion).\n\n```bash\npython -m easyhec.examples.sim.maniskill -e StackCube-v1 --samples 5 --seed 2 \\\n    --camera-name hand_camera \\\n    --initial-extrinsic-guess-rot-error 5 --initial-extrinsic-guess-pos-error 0.01\n```\n\nInstead of using the ground truth segmentation masks of the robot you can also use SAM2 generated masks based on your own point based annotations to reflect what you might do with real world images without access to ground truth data. The script below turns that option on and opens a GUI on your display to let you annotate each generated sample.\n\n```bash\npython -m easyhec.examples.sim.maniskill -e StackCube-v1 --samples 5 \\\n  --model-cfg ../sam2/configs/sam2.1/sam2.1_hiera_l.yaml --checkpoint ../sam2/checkpoints/sam2.1_hiera_large.pt \\\n  --no-use-ground-truth-segmentation\n```\n\n### Real\n\nlerobot example is WIP.\n\n## Customization\n\nThis repository is fairly minimal. The core optimization code is in `easyhec/optim` which uses a few 3D geometry related utilities in `easyhec/utils`. Feel free to copy those and modify as needed.\n\n## Tuning Tips\n\n- It is recommended to get a diverse range of sample images that show the robot in different orientations. This is particularly more important for wrist cameras, which often only see the robot gripper.\n- The initial guess of the camera extrinsics does not have be good, but if the robot is up very close it may need to be more accurate. This can be the case for wrist cameras.\n- To ensure best results make sure you have fairly accurate visual meshes for the robot the camera is attached on / is pointing at. It is okay if the colors do not match, just the shapes need to match.\n- While it is best to have accurate visual meshes, this optimization can still work even if you don't include some parts from the real world. It may be useful to edit out poor segmentations.\n- It is okay if the loss is in the 1000s and does not go down. Loss values do not really reflect the accuracy of the extrinsic estimate since it can depend on camera resolution and how far away the robot is.\n\n## Citation\n\nIf you find this code useful for your research, please use the following BibTeX entries\n\n```\n@article{chen2023easyhec,\n  title={EasyHec: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration},\n  author={Chen, Linghao and Qin, Yuzhe and Zhou, Xiaowei and Su, Hao},\n  journal={IEEE Robotics and Automation Letters (RA-L)}, \n  year={2023}\n}\n@article{Laine2020diffrast,\n  title   = {Modular Primitives for High-Performance Differentiable Rendering},\n  author  = {Samuli Laine and Janne Hellsten and Tero Karras and Yeongho Seol and Jaakko Lehtinen and Timo Aila},\n  journal = {ACM Transactions on Graphics},\n  year    = {2020},\n  volume  = {39},\n  number  = {6}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstonet2000%2Fsimple-easyhec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstonet2000%2Fsimple-easyhec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstonet2000%2Fsimple-easyhec/lists"}