{"id":22433424,"url":"https://github.com/nepfaff/sim2sim","last_synced_at":"2026-02-25T22:03:21.565Z","repository":{"id":265541158,"uuid":"569105235","full_name":"nepfaff/sim2sim","owner":"nepfaff","description":"A simulation-based evaluation system for real2sim approaches","archived":false,"fork":false,"pushed_at":"2024-09-19T15:06:03.000Z","size":37855,"stargazers_count":2,"open_issues_count":9,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-29T20:55:48.047Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nepfaff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-22T04:53:23.000Z","updated_at":"2024-11-02T04:18:03.000Z","dependencies_parsed_at":"2024-11-29T20:56:30.543Z","dependency_job_id":"07b57ba6-e178-49db-8816-df2ffb6bf180","html_url":"https://github.com/nepfaff/sim2sim","commit_stats":null,"previous_names":["nepfaff/sim2sim"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nepfaff%2Fsim2sim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nepfaff%2Fsim2sim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nepfaff%2Fsim2sim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nepfaff%2Fsim2sim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nepfaff","download_url":"https://codeload.github.com/nepfaff/sim2sim/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228377561,"owners_count":17910449,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-05T22:14:53.860Z","updated_at":"2026-02-25T22:03:16.546Z","avatar_url":"https://github.com/nepfaff.png","language":"Python","readme":"# sim2sim\nA simulation-based evaluation system for real2sim approaches\n\nA high-level system diagram is shown below:\n![system_overview_diagram](system_overview_diagram.png)\n\n## Installation\n\nThis repo uses Poetry for dependency management. To setup this project, first install\n[Poetry](https://python-poetry.org/docs/#installation) and, make sure to have Python3.10\ninstalled on your system.\n\nThen, configure poetry to setup a virtual environment that uses Python 3.10:\n```\npoetry env use python3.10\n```\n\nNext, install all the required dependencies to the virtual environment with the\nfollowing command:\n```\npoetry install -vvv\n```\n(the `-vvv` flag adds verbose output).\n\nFor local Drake and manipulation installations, insert the following at the end of the\n`.venv/bin/activate` and `.venv/bin/activate.nu` files, modifying the paths and python version as required:\n```bash\nexport PYTHONPATH=~/drake-build/install/lib/python3.10/site-packages:${PYTHONPATH}\nexport PYTHONPATH=~/manipulation:${PYTHONPATH}\n```\n\nActivate the environment:\n```\npoetry shell\n```\n\nInstall `git-lfs`:\n\n```bash\ngit-lfs install\ngit-lfs pull\n```\n\nOptionally install the [learning_real2sim](https://github.com/liruiw/learning_real2sim) repo in development mode based on the\ninstructions in its `README`. Note that this repo is only needed for a small subset of experiments.\n\n### Requirements for using mesh decomposition\n\n1. Install [pointnet-pytorch](https://github.com/liruiw/Pointnet2_PyTorch).\n2. Install [v-hacd](https://github.com/mikedh/trimesh/blob/30a423b884903905aba82408255f02dec0b33175/docker/builds/vhacd.bash)\nin trimesh by running the script.\n\n### Requirement for compliant meshes\n\nInstall [fTetWild](https://github.com/wildmeshing/fTetWild) and move the executable `FloatTetwild_bin` to a `bin` directory.\n\n## Experiment Types\n\nThis repo provides two different experiment types. The first one is the sim2sim pipeline experiment as shown in the\nsystem overview diagram. The second one is for comparing two different sim2sim pipelines by running one of them as the\n'outer' and one of them as the 'inner'. The main difference here is that the sim2sim pipeline is duplicated and one of\nthem is run instead of the hand-crafted 'outer'/ real-world manipuland. This is particularly useful for evaluating how\ntwo different sim2sim pipelines differ from each other (e.g. by using the contact force visualizer).\n\nThe sim2sim pipeline comparison experiment type is shown in the image below:\n![pipeline_comparison_diagram](pipeline_comparison_diagram.png)\n\n### Experiment description files\n\nAn experiment description file deterministically specifies an experiment. New experiments can easily be constructed by\nmixing components and their parameters in an experiment description file.\n\nAn example of a sim2sim pipeline experiment can be found in `experiments/planar_pushing/box/coacd.yaml` and\nan example of a sim2sim pipeline comparison experiment can be found in\n`experiments/planar_pushing/sphere/coacd_vs_hydroelastic_coacd.yaml`. These two look very similar apart from the\n`is_pipeline_comparison` parameter and that there is an `inner` and `outer` version for most components in the pipeline\ncomparison experiment.\n\n## Running an experiment\n\nReplace `experiments/table_pid/table_pid_simple.yaml` in the command below with your experiment description file.\nThe experiment description file deterministically specifies an experiment.\n\n```bash\npython scripts/run_experiment.py --experiment_description experiments/planar_pushing/box/coacd.yaml\n```\n\n## Contact Force Visualizer\n\nThe visualizer can be used to visualize both `outer` and `inner` manipulands and their contact forces over time.\n\nExample usage:\n```bash\npython3 scripts/visualize_contact_forces.py --data logs/sphere_pushing_coacd/ --separation_distance 0.2\n```\n\nArrow colors:\n- ![#0000FF](https://placehold.co/15x15/0000FF/0000FF.png) Outer generalized contact force (force component)\n- ![#9999FF](https://placehold.co/15x15/9999FF/9999FF.png) Outer generalized contact force (torque component)\n- ![#FF0080](https://placehold.co/15x15/FF0080/FF0080.png) Inner generalized contact force (force component)\n- ![#FF99CC](https://placehold.co/15x15/FF99CC/FF99CC.png) Inner generalized contact force (torque component)\n- ![#00FF00](https://placehold.co/15x15/00FF00/00FF00.png) Outer point/ hydroelastic contact force\n- ![#00FFFF](https://placehold.co/15x15/00FFFF/00FFFF.png) Outer hydroelastic contact torque\n- ![#FF0000](https://placehold.co/15x15/FF0000/FF0000.png) Inner point/ hydroelastic contact force\n- ![#FF8000](https://placehold.co/15x15/FF8000/FF8000.png) Inner hydroelastic contact torque\n\nIt is possible to step through time using the left and right arrow keys. The \"toggle\" buttons in meshcat can be used to\nshow and hide items by default when stepping through time.\nSee `scripts/visualize_contact_forces.py` for all available arguments.\n\n## Evaluating real2sim approaches quantitatively\n\n### Evaluation based on errors at final timestep\n\nThe following script can be used for ranking different real2sim approaches based on translation, rotation, and velocity\nerrors at the final timestep.\n\n```bash\npython scripts/rank_real2sim_approaches_final_errors.py --experiment_descriptions experiments/floor_drop/\n```\n\nwhere `experiments/floor_drop/` is a folder containing experiment description files.\n\nThe intended setup are experiment description files that describe the same experiment but with different real2sim\napproaches. For example, they may have different real2sim pipeline components or different component parameters.\nThe experiment description files are ranked based on a weighted combination of the individual errors. The weights are\nhardcoded in `scripts/rank_real2sim_approaches_final_errors.py`.\n\n### Evaluating a primitive representation collection based on errors at the final timestep\n\nThe following script can be used for ranking representations in a primitive representation based on translation, rotation,\nand velocity errors at the final timestep.\n\n```bash\npython scripts/evaluate_primitive_representation_collection.py --paths '[\"absolute_path/to/representation_collection/\"]' --experiment_description experiments/planar_pushing/box/mustard_raw_tsdf_vs_spheres_equation_error.yaml --eval_contact_model\n```\n\nwhere `representation_collection/` is a folder with the following structure:\n- representation_collection/\n    - representation1/\n        - primitive_info.pkl\n    - representation2/\n        - primitive_info.pkl\n    - ...\n    - representationN/\n        - primitive_info.pkl\n    - physical_properties.yaml\n\n`physical_properties.yaml` must contain a `mass`, `inertia`, and `com` field.\n\nThe experiment description specifies the experiment and must contain the following components:\n- `IdentityPrimitiveMeshProcessor` for `inner_mesh_processor`\n- `GTPhysicalPropertyEstimator` for `outer_physical_property_estimator`\n- `GTPhysicalPropertyEstimator` for `inner_physical_property_estimator`\n- `IdentityInverseGraphics` for `inner_inverse_graphics` with a valid `mesh_pose` if using the `--additional_collision_geometries_path` argument\n\n**NOTE:** Make sure that `inner_env` uses the desired contact model for simulating the representation collection (`point` or `hydroelastic_with_fallback`).\n\n## Generating a mesh dynamic distance dataset\n\n**NOTE:** Currently only the `random_force` experiment is supported.\n\n1. Specify the desired experiment parameters in `experiments/random_force/random_force_metaball.yaml`.\n2. Generate the data using one of the following commands (with your arguments):\n    ```bash\n    python scripts/collect_random_force_data.py --experiment_description experiments/random_force/random_force_gmm.yaml --logging_path logs/metric_learning_data --num_runs_per_perturbation 10 --num_perturbations 1000\n    ```\n    ```bash\n    python3 scripts/collect_planar_pushing_data.py --experiment_description experiments/planar_pushing/sphere/gmm.yaml --logging_path logs/metric_learning_data --num_runs_per_perturbation 10 --num_perturbations 1000\n    ```\n3. Postprocess the data:\n    ```bash\n    python scripts/postprocess_metric_learning_dataset.py --data_path logs/metric_learning_data\n    ```\n    \n### Generating a metric for a specific representation\n\n1. Define the representation/ mesh processor in a `random_force.yaml` file.\n2. Generate the data using the following command (with your arguments):\n    ```bash\n    python scripts/collect_representation_specific_random_force_data.py --experiment_description experiments/random_force/random_force.yaml --logging_path logs/mean_random_force --num_runs 50\n    ```\n3. Postprocess the data:\n    ```bash\n    python scripts/postprocess_metric_learning_dataset.py --is_representation_specific --data_path logs/mean_random_force\n    ```\n    \n## Generating simulator timing scale data with number of ellipsoids\n\n1. Comment out all meshcat specific stuff in `simulation/random_force_simulator.py` (Drake crashes if more than 100\nmeshcats are spawned and it is not possible to clean them up without terminating the top-level script until [this issue](https://github.com/RobotLocomotion/drake/issues/19362) is resolved).\n2. Collect the data:\n    ```bash\n    python scripts/collect_simulation_complexity_data.py --logging_path logs/simulation_complexity --experiment_description experiments/random_force/random_force_gmm.yaml\n    ```\n3. Create the plot in `logs/simulation_complexity':\n    ```bash\n    python scripts/postprocess_simulation_complexity_data.py --data_path logs/simulation_complexity/\n    ```\n    \n## Data format\n\nThe camera extrinsics are homogenous `world2cam` transformation matrices with `OpenCV convention`.\n\n## Error: \"Meshes does not have textures\"\n\nThis requires modifying the pytorch3d python files as suggested [here](https://github.com/facebookresearch/pytorch3d/issues/333#issuecomment-678129430) (remember to do it for all the shaders, the line numbers are no longer accurate).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnepfaff%2Fsim2sim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnepfaff%2Fsim2sim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnepfaff%2Fsim2sim/lists"}