{"id":31355171,"url":"https://github.com/leggedrobotics/fdm","last_synced_at":"2025-09-26T16:50:34.494Z","repository":{"id":309339295,"uuid":"970696213","full_name":"leggedrobotics/fdm","owner":"leggedrobotics","description":"Learned Perceptive Forward Dynamics Model","archived":false,"fork":false,"pushed_at":"2025-08-11T09:27:47.000Z","size":536,"stargazers_count":181,"open_issues_count":1,"forks_count":4,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-08-11T11:30:31.815Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://leggedrobotics.github.io/fdm.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leggedrobotics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-22T11:58:17.000Z","updated_at":"2025-08-11T09:27:50.000Z","dependencies_parsed_at":"2025-08-11T11:30:36.967Z","dependency_job_id":"d3147162-ce16-44cf-a666-07fb0559ff02","html_url":"https://github.com/leggedrobotics/fdm","commit_stats":null,"previous_names":["leggedrobotics/fdm"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/leggedrobotics/fdm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Ffdm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Ffdm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Ffdm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Ffdm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leggedrobotics","download_url":"https://codeload.github.com/leggedrobotics/fdm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Ffdm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277112105,"owners_count":25762969,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-26T02:00:09.010Z","response_time":78,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-26T16:50:33.218Z","updated_at":"2025-09-26T16:50:34.484Z","avatar_url":"https://github.com/leggedrobotics.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n  Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation\n  \u003cbr\u003e\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n\n  [![Youtube Video](./docs/overview.png)](https://youtu.be/QqK0IKINJEQ?t=761)\n\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://leggedrobotics.github.io/fdm.github.io/\"\u003eProject Page\u003c/a\u003e •\n  \u003ca href=\"https://arxiv.org/abs/2504.19322\"\u003earXiv\u003c/a\u003e •\n  \u003ca href=\"https://youtu.be/QqK0IKINJEQ?t=761\"\u003eVideo\u003c/a\u003e •\n  \u003ca href=\"#citation\"\u003eBibTeX\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\n[![IsaacSim](https://img.shields.io/badge/IsaacSim-4.5.0-silver.svg)](https://docs.isaacsim.omniverse.nvidia.com/latest/index.html)\n[![Isaac Lab](https://img.shields.io/badge/IsaacLab-2.2.0-silver)](https://isaac-sim.github.io/IsaacLab)\n[![Python](https://img.shields.io/badge/python-3.10-blue.svg)](https://docs.python.org/3/whatsnew/3.10.html)\n[![Linux platform](https://img.shields.io/badge/platform-linux--64-orange.svg)](https://releases.ubuntu.com/20.04/)\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)](https://pre-commit.com/)\n[![License](https://img.shields.io/badge/license-BSD--3-yellow.svg)](https://opensource.org/licenses/BSD-3-Clause)\n\n\u003c/p\u003e\n\nOur novel perceptive Forward Dynamics Model (FDM) enables real-time, learned traversability assessment for safe robot navigation by predicting future states based on environmental geometry and proprioceptive history. Trained in simulation and fine-tuned with real-world data, the model captures the full system dynamics beyond rigid body simulation. Integrated into a zero-shot Model Predictive Path Integral (MPPI) planner, our approach removes the need for tedious cost function tuning, improving safety and generalization. Tested on the ANYmal legged robot, our method significantly boosts navigation success in rough environments, with effective sim-to-real transfer.\n\n⭐ If you find our perceptive FDM useful, star it on GitHub to get notified of new releases! The repository features:\n- Implementation of the perceptive FDM training code as extension for [IsaacLab](https://github.com/isaac-sim/IsaacLab)\n- Integration of the perceptive FDM into a [Model Predictive Path Integral (MPPI)](https://ieeexplore.ieee.org/document/7989202) planner\n- Real-world deployment of the perceptive FDM on the [ANYmal robot](https://rsl.ethz.ch/robots-media/anymal.html)\n\n\n## Paper\n\nA technical introduction to the theory behind our perceptive FDM is provided in our open-access RSS paper, available [here](https://arxiv.org/abs/2504.19322). For a quick overview, watch the accompanying 5-minute presentation [YouTube](https://youtu.be/QqK0IKINJEQ?t=761). More information about the work is available in the abstract below.\n\n\u003cdetails\u003e\n\u003csummary\u003eAbstract\u003c/summary\u003e\n\u003cbr\u003e\nEnsuring safe navigation in complex environments requires accurate real-time traversability assessment and understanding of environmental interactions relative to the robot's capabilities. Traditional methods, which assume simplified dynamics, often require designing and tuning cost functions to safely guide paths or actions toward the goal. This process is tedious, environment-dependent, and not generalizable. To overcome these issues, we propose a novel learned perceptive Forward Dynamics Model (FDM) that predicts the robot's future state conditioned on the surrounding geometry and history of proprioceptive measurements, proposing a more scalable, safer, and heuristic-free solution. The FDM is trained on multiple years of simulated navigation experience, including high-risk maneuvers, and real-world interactions to incorporate the full system dynamics beyond rigid body simulation. We integrate our perceptive FDM into a zero-shot Model Predictive Path Integral (MPPI) planning framework, leveraging the learned mapping between actions, future states, and failure probability. This allows for optimizing a simplified cost function, eliminating the need for extensive cost-tuning to ensure safety. On the legged robot ANYmal, the proposed perceptive FDM improves the position estimation by on average 41% over competitive baselines, which translates into a 27% higher navigation success rate in rough simulation environments. Moreover, we demonstrate effective sim-to-real transfer and showcase the benefit of training on synthetic and real data.\n\u003c/details\u003e\n\n### Citation\n```\n@inproceedings{roth2025fdm,\n  title={Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation},\n  author={Roth, Pascal and Frey, Jonas and Cadena, Cesar and Hutter, Marco},\n  booktitle={Robotics: Science and Systems (RSS 2025)},\n  year={2025}\n}\n```\n\n## Installation\n\n### IsaacLab Extension (Training and Evaluation)\n\n\u003cp style=\"background-color:#ffdddd; border-left:4px solid #f44336; padding:10px;\"\u003e\n  \u003cstrong\u003e⚠️ WARNING:\u003c/strong\u003e With our code update to the latest IsaacLab version, we experience the robot sinking into the ground which will affect model learning. We are investigating the issue.\n\u003c/p\u003e\n\nThe extension is developed with [IsaacLab version 2.1.1](https://github.com/isaac-sim/IsaacLab/tree/v2.1.1) (latest\ntested commit 19b24c7). Future versions may work, but are not tested. IsaacLab runs on Ubuntu 20.04 - 24.04. Our paper\nresults are using the locomotion policy by [Miki et al.](https://www.science.org/doi/full/10.1126/scirobotics.abk2822).\nAs this one is not public, the repos default is the standard IsaacLab ANYmal policy with reduced stability. Furthermore,\nno new parameter parameter tuning has been performed. Results may therefore differ.\n\nThe extension is installed as follows:\n\n1. **Install IsaacSim and IsaacLab:** \\\n   Follow the [IsaacLab installation guide](https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/index.html)\n   to ensure IsaacSim and IsaacLab are installed. \\\n   NOTE: Please use an IsaacLab version where [PR2183](https://github.com/isaac-sim/IsaacLab/pull/2183) has been merged, which contains changes necessary to run the scripts successfully.\n\n2. **Clone this repository and install as part of IsaacLab:**\n\n   ```bash\n   git clone git@github.com:leggedrobotics/fdm.git\n\n   cd \u003cpath-to-isaaclab-repo\u003e\n\n   ./isaaclab.sh -p -m pip install -e \u003cpath-to-your-fdm-repo\u003e/exts/fdm\n   ./isaaclab.sh -p -m pip install -e \u003cpath-to-your-fdm-repo\u003e/nav-suite/exts/nav_suite\n   ./isaaclab.sh -p -m pip install -e \u003cpath-to-your-fdm-repo\u003e/nav-suite/exts/nav_tasks\n\n   ```\n   **Important**: Make sure the submodules are correctly initialized and assets are downloaded from git lfs (Install instructions [here](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage)):\n   ```bash\n   cd \u003cpath-to-your-fdm-repo\u003e\n   git submodule update --init\n   git lfs pull\n   ```\n\n3. **Verify the installation:**\n   To verify the installation, run the training script in debug mode:\n   ```bash\n   cd \u003cpath-to-your-fdm-repo\u003e\n   \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/train.py --mode debug\n   ```\n   The IsaacSim GUI should open and data collection should start, meaning the robots are moving. Later, the FDM training should start.\n\nFor further details on the IsaacLab extensions, see the [README](exts/fdm/docs/README.md)\n\n### ROS Integration (Real-World Deployment)\n\nThe integration is done with ROS Noetic on Ubuntu 20.04 for ANYmal D (Release 24.04). For details on the ROS integration, see the [README](ros/README.md).\n\n1. **Create a catkin workspace:**\n   ```bash\n   mkdir -p fdm_ws/src\n   cd fdm_ws/src\n   ```\n\n2. **Clone the Repository:**\n   ```bash\n   git clone https://github.com/leggedrobotics/forward_dynamics_model.git\n   ```\n\n3. **Build the workspace:**\n   ```bash\n   cd ..\n   catkin build fdm_navigation_ros waypoint_rviz_plugin\n   ```\n\n4. **Install the FDM Navigation Package**\n   ```bash\n   pip install -e src/forward_dynamics_model/forward_dynamics_model/ros/\n   ```\n\n4. **Verify the installation:**\n\n   Source the workspace and execute the planner.\n   ```bash\n   source devel/setup.bash\n   roslaunch fdm_navigation_ros planner.launch\n   ```\n\nIn order to use the `WaypointTool` to set waypoints for the planner, it has to be added to the rviz gui of ANYbotics.\nTo do so, add following line under `Visualization Manager -\u003e Tools`:\n\n```.rviz\nVisualization Manager:\n   ...\n   Tools:\n      - Class: rviz/WaypointTool\n        Topic: waypoint\n```\n\n\n## Usage\n\n### Model Demo\n\nThe latest model (`model.zip`) trained with the locomotion policy of Miki et al. are available to download:\n- [Simulation Model](https://drive.google.com/file/d/1_vkiW4cwW7f--ua7N7ZO5LR8-eOF3Zga/view?usp=sharing)\n- [Real-World Fine-Tuned Model](https://drive.google.com/file/d/1mphOSa1ar3sw4IfVPSEcUnGXEN7wiBU9/view?usp=sharing)\n\n**IMPORTANT** For evaluations in simulation, it is recommended to use the simulation model as it is not fitted for a particular real-world platform.\n\n**NOTE**: The repos default is the IsaacLab locomotion policy, as the one of Miki et al. is not public. For comparable results, please retrain with possible adjusted parameters.\n\nFollow the instructions above to setup the FDM extension for IsaacLab, then extract the model inside `\u003cpath-to-your-fdm-repo\u003e/logs/fdm/fdm_se2_prediction_depth/fdm_latest`:\n\n```bash\nmkdir -p \u003cpath-to-your-fdm-repo\u003e/logs/fdm/fdm_se2_prediction_depth/fdm_latest\ncd \u003cpath-to-your-fdm-repo\u003e/logs/fdm/fdm_se2_prediction_depth/fdm_latest\nmv \u003cpath-to-model-download\u003e/model.zip .\nunzip model.zip\n```\n\nthen run the test script for the dynamics estimation model:\n\n```bash\ncd \u003cpath-to-your-fdm-repo\u003e\n\u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/test.py --runs fdm_latest\n```\n\nor for the planning first set the parameters into `planning_mode` (during training some bounds are higher to increase\nstability later but then are adjusted for the actual planning tasks), this can be done in the\n[__init__](./exts/fdm/fdm/__init__.py) line 36. The planner test script can then be executed as follows:\n\n```bash\ncd \u003cpath-to-your-fdm-repo\u003e\n\u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/plan_test.py --run fdm_latest --mode test\n```\n\nTo get the same simulation plot as provided in the paper, execute\n\n```bash\n./isaaclab.sh -p fdm_sub/scripts/plan_test.py --run fdm_latest --mode plot\n```\n\n### Training\n\nTo train the Forward Dynamics Model, follow these steps:\n\n1. **Simulation Pre-Training:**\n   Run the training script:\n\n   ```bash\n   \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/train.py --mode train  --run_name \u003cyour-run-name\u003e\n   ```\n\n   This will per default execute a training on ANYmal with a perceptive policy. For arguments that can be passed directly to the run script, see `\u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/train.py -h`. Detailed configuration for the environment, observation space, model, and much more is given in:\n\n   - [IsaacLab Configs (mdp, terrain, robot, ...)](exts/fdm/fdm/env_cfg)\n   - [Model Configs](exts/fdm/fdm/model)\n   - [Trainer/ Runner Configs (optimizer, learning rate, collection intervals)](exts/fdm/fdm/runner)\n\n   **Note**: For wandb logging during training, you need to set the following environment variables:\n   - `WANDB_API_KEY`: Your Weights \u0026 Biases API key (required)\n   - `WANDB_ENTITY`: Your Weights \u0026 Biases entity/username (can also be set in [TrainerCfg](exts/fdm/fdm/runner/trainer/trainer_cfg.py))\n   - `WANDB_MODE`: Wandb mode, defaults to \"online\" if not set (can also be set in [TrainerCfg](exts/fdm/fdm/runner/trainer/trainer_cfg.py))\n\n   During the training, evaluation steps in test environments can be performed. The data for those environments in typically pre-collected and remains constant for different trainings. The dataset paths can be defined in the [TrainerCfg](exts/fdm/fdm/runner/trainer/trainer_cfg.py). The collected of the datasets is done by executing:\n\n   ```bash\n   \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/test_data_collector.py --test_env \u003cenv-name\u003e\n   ```\n\n2. **Real-World Data Collection:**\n\n   You need to extract the data from the collected rosbags. The scripy relies on ROS Noetic, so it has to be executed with python 3.8. The script has been tested with the leggedrobot ANYmal (it likely does not work with other robots).\n\n   ```bash\n   python ros/rosbag_tools/rosbag_extractor.py \\\n      -bf \u003cpath_to_rosbag\u003e \\\n      -o \u003cpath_to_output_dir\u003e \\\n      --unified_frame odom \\\n      -t \"/elevation_mapping/elevation_map_raw\" \"/anymal_low_level_controller/actuator_readings\" \"/state_estimator/anymal_state\" \"/state_estimator/odometry\" \"/twist_mux/twist\"\n   ```\n\n   This extracts the data from the rosbag. To bring the data into the format required by the FDM, run the following script which at the same time will evaluate your current model on the collected data. Please adjust the configs inside the script to match the extracted data.\n\n   ```bash\n   ./isaaclab.sh -p scripts/real_world_test.py\n   ```\n\n   This process creates `train.pkl` and `val.pkl` files, used for fine tuning.\n   The fine-tuning data used in this work is available [here](https://drive.google.com/file/d/1aFuScInnLjh2eukqOnnL-5ydqcUSpUbE/view?usp=sharing) and should be unzipped to `IsaacLab/logs/fdm/real_world_datasets`.\n\n   The original rosbags are made available as part of the RSL GrandTour Dataset see [here](https://grand-tour.leggedrobotics.com/).\n\n3. **Real-World Fine-Tuning:**\n\n   To fine-tune the model on the collected data, run the following script:\n\n   ```bash\n   \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/train.py --mode train-real-world  --run_name \u003cyour-run-name\u003e  --real-world-dilution 10\n   ```\n\n   The real-world datasets are currently hard-coded in the `train.py` function, to the ones that can be downloaded as described above. Please adjust the paths for your own data in the `train.py` script.\n\n### Evaluation\n\n1. **Dynamics Estimation**\n\n   - **Qualitative** evaluation\n\n      A qualitative evaluation of the dynamics estimation in **defined test environments** can be executed using the following command. This executed as many commands as the prediction horizon is long and then stops the robot to then give time to closly evaluate the predictions.\n\n      ```bash\n      \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/test.py --runs \u003crun-name\u003e\n      ```\n\n      Given the argument `--paper-figure` generates Fig. 4 of the paper. Executing the script with `--paper-platform-figure` and varying the platforms using `--robot` argument generates Fig. 7.\n\n      **IMPORTANT**: This script exports the policies to `jit` format and therefore has to be executed before the execution of the ROS code.\n\n      A qualitative evaluation in **any environment** can be executed using the following command. Here longer trajectories are executed and the predictions are shown continuously along them.\n\n      ```bash\n      \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/eval.py --runs \u003crun-name\u003e --terrain-cfg \u003cterrain-name\u003e\n      ```\n\n      Possible terrain names are defined in the terrain config in [TerrainConfigs](exts/fdm/fdm/env_cfg/terrain_cfg.py).\n\n   - **Quantitative** evaluation\n\n      A **quantitative** evaluation between the constant velocity assumption, the baseline method by Kim et al., and the developed method is executed with the following command:\n\n      ```bash\n      \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/eval_metrics.py --runs \u003crun-name\u003e\n      ```\n\n   - **Real-World** evaluation\n\n      Evaluation on the real-world data is done by running:\n\n      ```bash\n      \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/real_world_eval_metrics.py --runs \u003crun-name\u003e\n      ```\n\n2. **Planning**\n\n   The planning evaluation is done by calling:\n\n   ```bash\n   \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/plan_test.py --run \u003crun-name\u003e\n   ```\n\n   The script runs with different modes:\n   - `--mode test`: **qualitative** evaluation in a test environment\n   - `--mode metric --env_type 2D`: **qualitative** evaluation in a 2D environment\n   - `--mode metric --env_type 3D`: **qualitative** evaluation in a 3D environment\n   - `--mode plot`: generate the Fig. 9 of the paper\n\n   The predicted paths and their rewards can be visualized using `--cost_show` argument.\n\n   While this function uses pre-defined environments with start-goal pairs selected so that the planner has to overcome/ avoid certain obstacles, a evaluation with random start-goal pairs can be executed by running:\n\n    ```bash\n   \u003cpath-to-isaaclab-repo\u003e/isaaclab.sh -p scripts/plan_metric.py --run \u003crun-name\u003e\n   ```\n\n   As this involves many straight paths without significant obstacles, the `plan_test.py` script provide a more representative evaluation.\n   For planning without\n\n### Real-World Deployment\n\nPolicies have to be `jit` exported before running the ROS code. The export is performed when running the `test.py` script as described above.\nGiven the export, adjust the path in the [config file](ros/fdm_navigation_ros/config/default.yaml).\nThen execute the following steps to run the planner:\n\n1. **Source the workspace:**\n   ```bash\n   source devel/setup.bash\n   ```\n\n2. **Run the planner:**\n   ```bash\n   roslaunch fdm_navigation_ros planner.launch\n   ```\n\n## Contributing\n\nContributions are welcome! Please ensure that your code adheres to the project's coding standards by running the defined pre-commit formatter.\nPlease note that due to different python version of ROS (python 3.8) and IsaacLab (\u003e python 3.10), both have to be installed on the system.\n\n```bash\npre-commit run --all-files\n```\n\n\n### License\n\nThis code belongs to Robotic Systems Lab, ETH Zurich.\nAll rights reserved.\n\n**Authors: [Pascal Roth](https://pascal-roth.github.io/), [Jonas Frey](https://jonasfrey96.github.io/), [Cesar Cadena](https://scholar.google.ch/citations?hl=en\u0026user=aOns5HQAAAAJ), and [Marco Hutter](https://scholar.google.ch/citations?user=DO3quJYAAAAJ\u0026hl=en)\u003cbr /\u003e\nMaintainer: Pascal Roth, rothpa@ethz.ch**\n\nThe FDM inference part (ROS directory) has been tested under ROS Noetic on Ubuntu 20.04.\nThe training code has been tested in Ubuntu 24.04.\nThis is research code, expect that it changes often and any fitness for a particular purpose is disclaimed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleggedrobotics%2Ffdm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleggedrobotics%2Ffdm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleggedrobotics%2Ffdm/lists"}