{"id":23111744,"url":"https://github.com/tim-salzmann/l4casadi","last_synced_at":"2025-04-05T17:07:24.361Z","repository":{"id":176548442,"uuid":"655767873","full_name":"Tim-Salzmann/l4casadi","owner":"Tim-Salzmann","description":"Use PyTorch Models with CasADi and Acados in Python, C(++) or Matlab","archived":false,"fork":false,"pushed_at":"2024-05-22T12:22:59.000Z","size":25795,"stargazers_count":251,"open_issues_count":0,"forks_count":16,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-05-22T12:33:41.479Z","etag":null,"topics":["acados","casadi","cplusplus","deep-learning","learning-control","machine-learning","optimization","optimization-algorithms","python","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tim-Salzmann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-19T14:55:52.000Z","updated_at":"2024-05-29T09:27:23.748Z","dependencies_parsed_at":"2023-10-05T02:15:42.609Z","dependency_job_id":"6249d64c-4231-4698-b82a-77f45f5c0c81","html_url":"https://github.com/Tim-Salzmann/l4casadi","commit_stats":null,"previous_names":["tim-salzmann/l4casadi"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tim-Salzmann%2Fl4casadi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tim-Salzmann%2Fl4casadi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tim-Salzmann%2Fl4casadi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tim-Salzmann%2Fl4casadi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tim-Salzmann","download_url":"https://codeload.github.com/Tim-Salzmann/l4casadi/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247369952,"owners_count":20927928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acados","casadi","cplusplus","deep-learning","learning-control","machine-learning","optimization","optimization-algorithms","python","pytorch"],"created_at":"2024-12-17T02:10:12.865Z","updated_at":"2025-04-05T17:07:24.340Z","avatar_url":"https://github.com/Tim-Salzmann.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI version](https://badge.fury.io/py/l4casadi.svg)](https://badge.fury.io/py/l4casadi)\n![L4CasADi CI](https://github.com/Tim-Salzmann/l4casadi/actions/workflows/ci.yaml/badge.svg)\n![Downloads](https://img.shields.io/pypi/dm/l4casadi.svg)\n\n---\n# Learning 4 CasADi Framework\n\nL4CasADi enables the seamless integration of PyTorch-learned models with CasADi for efficient and potentially\nhardware-accelerated numerical optimization.\nThe only requirement on the PyTorch model is to be traceable and differentiable.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./examples/nerf_trajectory_optimization/media/animation.gif\" alt=\"Collision-free minimum snap optimized trajectory through a NeRF\" width=\"200\" height=\"200\"\u003e\n  \u003cimg src=\"./examples/fish_turbulent_flow/media/trajectory_generation_vorticity.gif\" alt=\"Energy Efficient Fish Navigation in Turbulent Flow\" width=\"400\" height=\"200\"\u003e\n  \u003cbr\u003e\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/Tim-Salzmann/l4casadi/blob/main/examples/nerf_trajectory_optimization/NeRF_Trajectory_Optimization.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e\n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/Tim-Salzmann/l4casadi/blob/main/examples/fish_turbulent_flow/Fish_Turbulent_Flow.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e\n  \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n  \u003cp\u003e\u003ci\u003eTwo L4CasADi examples: Collision-free trajectory through a NeRF and Navigation in Turbulent Flow\u003c/i\u003e\u003c/p\u003e\n\u003c/div\u003e\n\narXiv: [Learning for CasADi: Data-driven Models in Numerical Optimization](https://arxiv.org/pdf/2312.05873.pdf)\n\nTalk: [Youtube](https://youtu.be/UYdkRnGr8eM?si=KEPcFEL9b7Vk2juI\u0026t=3348)\n\n## L4CasADi v2 Breaking Changes\nAfter feedback from first use-cases L4CasADi v2 is designed with efficiency and simplicity in mind.\n\nThis leads to the following breaking changes:\n\n- L4CasADi v2 can leverage PyTorch's batching capabilities for increased efficiency. When passing `batched=True`,\nL4CasADi will understand the **first** input dimension as batch dimension. Thus, first and second-order derivatives\nacross elements of this dimension are assumed to be **sparse-zero**. To make use of this, instead of having multiple calls to a L4CasADi function in\nyour CasADi program, batch all inputs together and have a single L4CasADi call. An example of this can be seen when\ncomparing the [non-batched NeRF example](examples/nerf_trajectory_optimization/nerf_trajectory_optimization.py) with the\n[batched NeRF example](examples/nerf_trajectory_optimization/nerf_trajectory_optimization_batched.py) which is faster by\na factor of 5-10x.\n- L4CasADi v2 will not change the shape of an input anymore as this was a source of confusion. The tensor forwarded to\nthe PyTorch model will resemble the **exact dimension** of the input variable by CasADi. You are responsible to make\nsure that the PyTorch model handles a **two-dimensional** input matrix! Accordingly, the parameter\n`model_expects_batch_dim` is removed.\n- By default, L4CasADi v2 will not provide the Hessian, but the Jacobian of the Adjoint. This is sufficient for most\nmany optimization problems. However, you can explicitly request the generation of the Hessian by passing\n`generate_jac_jac=True`.\n\n[//]: # (L4CasADi v2 can use the new **torch compile** functionality starting from PyTorch 2.4. By passing `scripting=False`. This\nwill lead to a longer compile time on first L4CasADi function call but will lead to a overall faster\nexecution. However, currently this functionality is experimental and not fully stable across all models. In the long\nterm there is a good chance this will become the default over scripting once the functionality is stabilized by the\nTorch developers.)\n\n## Table of Content\n- [Projects using L4CasADi](#projects-using-l4casadi)\n- [Installation](#installation)\n    - [Prerequisites](#prerequisites)\n    - [Windows Installation](#windows-installation)\n    - [Pip Install (CPU Only)](#pip-install-cpu-only)\n    - [From Source (CPU Only)](#from-source-cpu-only)\n    - [GPU (CUDA)](#gpu-cuda)\n- [Quick Start](#quick-start)\n- [Online Learning](#online-learning-and-updating)\n- [Naive L4CasADi](#naive-l4casadi) - Use this for small Multi Layer Perceptron Models.\n- [Real-time L4CasADi](#real-time-l4casadi) - Use this for fast MPC with Acados.\n- [Examples](#examples)\n\nIf you use this framework please cite the following two papers:\n\n```\n@article{salzmann2023neural,\n  title={Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms},\n  author={Salzmann, Tim and Kaufmann, Elia and Arrizabalaga, Jon and Pavone, Marco and Scaramuzza, Davide and Ryll, Markus},\n  journal={IEEE Robotics and Automation Letters},\n  doi={10.1109/LRA.2023.3246839},\n  year={2023}\n}\n```\n\n```\n@inproceedings{{salzmann2024l4casadi,\n  title={Learning for CasADi: Data-driven Models in Numerical Optimization},\n  author={Salzmann, Tim and Arrizabalaga, Jon and Andersson, Joel and Pavone, Marco and Ryll, Markus},\n  booktitle={Learning for Dynamics and Control Conference (L4DC)},\n  year={2024}\n}\n```\n\n## Projects using L4CasADi\n- Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms \u003cbr/\u003e [Paper](https://arxiv.org/pdf/2203.07747.pdf) | [Code](https://github.com/TUM-AAS/neural-mpc)\n- AC4MPC: Actor-Critic Reinforcement Learning for Nonlinear Model Predictive Control \u003cbr/\u003e [Paper](https://arxiv.org/pdf/2406.03995)\n- Reinforcement Learning based MPC with Neural Dynamical Models \u003cbr/\u003e [Paper](https://folk.ntnu.no/skoge/publications/2024/adhau-ecc24/ECC24_0903_FI.pdf)\n- Neural Potential Field for Obstacle-Aware Local Motion Planning \u003cbr/\u003e [Paper](https://arxiv.org/pdf/2310.16362.pdf) | [Video](https://www.youtube.com/watch?v=KL3bfvUwGqs) | [Code](https://github.com/cog-isa/NPField)\n- N-MPC for Deep Neural Network-Based Collision Avoidance exploiting Depth Images \u003cbr/\u003e [Paper](https://arxiv.org/pdf/2402.13038.pdf) | [Code](https://github.com/ntnu-arl/colpred_nmpc)\n- An Integrated Framework for Autonomous Driving Planning and Tracking based on NNMPC Considering Road Surface Variations \u003cbr/\u003e [Paper](https://ieeexplore.ieee.org/abstract/document/10571565)\n\nIf your project is using L4CasADi and you would like to be featured here, please reach out.\n\n---\n## Installation\n### Prerequisites\nIndependently if you install from source or via pip you will need to meet the following requirements:\n\n- Working build system: CMake compatible C++ compiler.\n    - **On Linux/macOS:** GCC version 10 or higher is recommended.\n    - **On Windows:** You must install CMake and **Visual C++ Build Tools** (do not use a GCC compiler).\n- PyTorch (`\u003e=2.0`) installation in your python environment.\\\n`python -c \"import torch; print(torch.__version__)\"`\n\n### Windows Installation\n\nFor users installing on Windows, please ensure the following steps are taken:\n\n1. **Install CMake and Visual C++ Build Tools:**\nDownload and install [CMake](https://cmake.org/download/) and the [Visual C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/).\n2. **Ensure Tools Are on Your PATH:**\nMake sure both CMake and the Visual C++ Build Tools are added to your system's PATH. This can be done either via the installers (by selecting the option to update the PATH) or manually.\n3. **Mind the File Path Length:**\nWhen building the L4CasADi model, ensure that the directory path does not exceed Windows’ maximum character limit. Long paths might lead to build issues, so choose a directory with a short path if possible.\n\n### Pip Install (CPU Only)\n- Ensure Torch CPU-version is installed\\\n`pip install torch\u003e=2.0 --index-url https://download.pytorch.org/whl/cpu`\n- Ensure all build dependencies are installed\n```\nsetuptools\u003e=68.1\nscikit-build\u003e=0.17\ncmake\u003e=3.27\nninja\u003e=1.11\n```\n\n- Run\\\n`pip install l4casadi --no-build-isolation`\n\n### From Source (CPU Only)\n- Clone the repository\\\n`git clone https://github.com/Tim-Salzmann/l4casadi.git`\n\n- All build dependencies installed via\\\n`pip install -r requirements_build.txt`\n\n- Build from source\\\n`pip install . --no-build-isolation`\n\nThe `--no-build-isolation` flag is required for L4CasADi to find and link against the installed PyTorch.\n\n### GPU (CUDA)\nCUDA installation requires nvcc to be installed which is part of the CUDA toolkit and can be installed on Linux via\n`sudo apt-get -y install cuda-toolkit-XX-X` (where `XX-X` is your installed Cuda version - e.g. `12-3`).\nOnce the CUDA toolkit is installed nvcc is commonly found at `/usr/local/cuda/bin/nvcc`.\n\nMake sure `nvcc -V` can be executed and run `pip install l4casadi --no-build-isolation` or `CUDACXX=\u003cPATH_TO_NVCC\u003e pip install . --no-build-isolation` to build from source.\n\nIf `nvcc` is not automatically part of your path you can specify the `nvcc` path for L4CasADi.\nE.g. `CUDACXX=\u003cPATH_TO_NVCC\u003e pip install l4casadi --no-build-isolation`.\n\n---\n\n## Quick Start\n\nDefining an L4CasADi model in Python given a pre-defined PyTorch model is as easy as\n```python\nimport l4casadi as l4c\n\nl4c_model = l4c.L4CasADi(pyTorch_model, device='cpu')\n```\n\nwhere the architecture of the PyTorch model is unrestricted and large models can be accelerated with dedicated hardware.\n\n---\n\n## Online Learning and Updating\nL4CasADi supports updating the PyTorch model online in the CasADi graph. To use this feature, pass `mutable=True` when\ninitializing a L4CasADi. To update the model, call the `update` function on the `L4CasADi` object.\nYou can optionally pass an updated model as parameter. If no model is passed, the reference passed at\ninitialization is assumed to be updated and will be used for the update.\n\n---\n\n## Naive L4CasADi\n\nWhile L4CasADi was designed with efficiency in mind by internally leveraging torch's C++ interface, this can still\nresult in overhead, which can be disproportionate for small, simple models. Thus, L4CasADi additionally provides a\n`NaiveL4CasADiModule` which directly recreates the PyTorch computational graph using CasADi operations and copies the\nweights --- leading to a pure C computational graph without context switches to torch. However, this approach is\nlimited to a small predefined subset of PyTorch operations --- only `MultiLayerPerceptron`\nmodels and CPU inference are supported.\n\nThe torch framework overhead dominates for networks smaller than three hidden layers, each with 64\nneurons (or equivalent). For models smaller than this size we recommend using the NaiveL4CasADiModule.\nFor larger models, the overhead becomes negligible and L4CasADi should be used.\n\nhttps://github.com/Tim-Salzmann/l4casadi/blob/f7b16fba90f4d3ee53217b560f26b47e6b23e44a/examples/naive/readme.py#L5-L9\n\n---\n## Real-time L4CasADi\nReal-time L4Casadi (former `Approximated` approach in [ML-CasADi](https://github.com/TUM-AAS/ml-casadi)) is the underlying framework powering\n[Real-time Neural-MPC](https://arxiv.org/pdf/2203.07747). It replaces complex models with local Taylor approximations.\nFor certain optimization procedures (such as MPC with multiple shooting nodes) this can lead to improved optimization times.\nHowever, `Real-time L4Casadi`, comes with many restrictions (only Python, no C(++) code generation, ...) and is therefore not\na one-to-one replacement for `L4Casadi`. Rather it is a complementary framework for certain special use cases.\n\nMore information [here](l4casadi/realtime).\n\nhttps://github.com/Tim-Salzmann/l4casadi/blob/f7b16fba90f4d3ee53217b560f26b47e6b23e44a/l4casadi/realtime/examples/readme.py#L32-L43\n\n---\n\n## Examples\nhttps://github.com/Tim-Salzmann/l4casadi/blob/f7b16fba90f4d3ee53217b560f26b47e6b23e44a/examples/readme.py#L28-L40\n\nPlease note that only `casadi.MX` symbolic variables are supported as input.\n\nMulti-input multi-output functions can be realized by concatenating the symbolic inputs when passing to the model and\nsplitting them inside the PyTorch function.\n\nTo use GPU (CUDA) simply pass `device=\"cuda\"` to the `L4CasADi` constructor.\n\nFurther examples:\n- Collision-free minimum snap optimized trajectory through a NeRF: [examples/nerf_trajectory_optimization](/examples/nerf_trajectory_optimization)\n- Energy Efficient Fish Navigation in Turbulent Flow: [examples/fish_turbulent_flow](/examples/fish_turbulent_flow)\n- Simple nonlinear programming with L4CasADi model as objective and constraints: [examples/simple_nlp.py](/examples/simple_nlp.py)\n- L4CasADi in pure C(++) projects: [examples/cpp_executable](/examples/cpp_usage)\n- Use PyTorch L4CasADi Model in Matlab: [examples/matlab](/examples/matlab)\n\n---\n\n## Acados Integration\nTo use this framework with Acados:\n- Follow the [installation instructions](https://docs.acados.org/installation/index.html).\n- Install the [Python Interface](https://docs.acados.org/python_interface/index.html).\n- Ensure that `LD_LIBRARY_PATH` is set correctly (`DYLD_LIBRARY_PATH`on MacOS).\n- Ensure that `ACADOS_SOURCE_DIR` is set correctly.\n\nAn example of how a PyTorch model can be used as dynamics model in the Acados framework for Model Predictive Control \ncan be found in [examples/acados.py](/examples/acados.py)\n\nTo use L4CasADi with Acados you will have to set `model_external_shared_lib_dir` and `model_external_shared_lib_name`\nin the `AcadosOcp.solver_options` accordingly.\n\n```\nocp.solver_options.model_external_shared_lib_dir = l4c_model.shared_lib_dir\nocp.solver_options.model_external_shared_lib_name = l4c_model.name\n```\n\nhttps://github.com/Tim-Salzmann/l4casadi/blob/f7b16fba90f4d3ee53217b560f26b47e6b23e44a/examples/acados.py#L156-L160\n\n---\n\n## FYIs\n\n\n### Warm Up\n\nNote that PyTorch builds the graph on first execution. Thus, the first call(s) to the CasADi function will be slow.\nYou can warm up to the execution graph by calling the generated CasADi function one or multiple times before using it.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftim-salzmann%2Fl4casadi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftim-salzmann%2Fl4casadi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftim-salzmann%2Fl4casadi/lists"}