https://github.com/borglab/swiftfusion
https://github.com/borglab/swiftfusion
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/borglab/swiftfusion
- Owner: borglab
- License: apache-2.0
- Created: 2019-11-10T19:36:21.000Z (over 6 years ago)
- Default Branch: main
- Last Pushed: 2022-12-18T21:24:57.000Z (over 3 years ago)
- Last Synced: 2025-05-08T21:45:58.056Z (over 1 year ago)
- Language: Swift
- Size: 7.62 MB
- Stars: 118
- Watchers: 13
- Forks: 13
- Open Issues: 43
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# SwiftFusion
A differentiable sensor fusion library written in Swift.
Think factor graphs (à la [GTSAM](https://gtsam.org)) coupled with deep learning (via [Swift for TensorFlow](https://www.tensorflow.org/swift)).
This project is in an early phase, but feel free to explore! Subject to *massive* changes. :-)
## Getting started
### Installation
#### macOS
Download a [Swift for TensorFlow macOS toolchain](https://github.com/tensorflow/swift/blob/main/Installation.md#development-snapshots) and follow the [macOS installation instructions](https://github.com/tensorflow/swift/blob/master/Installation.md#macos). Installing the latest development snapshot is recommended. [Check here](https://github.com/apple/swift/blob/main/docs/HowToGuides/GettingStarted.md#installing-dependencies) for the "required" version of Xcode for using nightly toolchains. Sometimes toolchains may also work with older Xcode minor versions than the one listed.
To use Swift for TensorFlow, add the toolchain to `PATH`:
```
export PATH="/Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/bin/:$PATH"
```
#### Linux
Download a [Swift for TensorFlow Linux toolchain](https://github.com/tensorflow/swift/blob/main/Installation.md#development-snapshots) and follow the [Linux installation instructions](https://github.com/tensorflow/swift/blob/master/Installation.md#linux). Installing the latest development snapshot is recommended.
#### NVIDIA Jetson
Download the [Swift for TensorFlow NVIDIA Jetson toolchain](https://storage.googleapis.com/swift-tensorflow-artifacts/oneoff-builds/swift-tensorflow-RELEASE-0.11-Jetson4.4.tar.gz) and follow the [Linux installation instructions](https://github.com/tensorflow/swift/blob/master/Installation.md#linux).
### Testing
```
swift test --enable-test-discovery
```
### Benchmarking
```
swift run -c release -Xswiftc -cross-module-optimization SwiftFusionBenchmarks
```
## Development
### Xcode (macOS)
Xcode provides a great Swift development experience on macOS.
To open SwiftFusion in Xcode, run `open Package.swift`.
### Visual Studio Code
To enable Swift autocomplete in Visual Studio Code, install the [Maintained Swift Development Environment plugin](https://marketplace.visualstudio.com/items?itemName=vknabel.vscode-swift-development-environment), and set the following plugin settings:
```
"sde.languageServerMode": "sourcekit-lsp",
"sourcekit-lsp.serverPath": "/usr/bin/sourcekit-lsp",
"sourcekit-lsp.toolchainPath": "",
"swift.path.swift_driver_bin": "/usr/bin/swift",
```
The [CodeLLDB plugin](https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb) enables debugging within Visual Studio Code. You need to set the following setting:
```
"lldb.library": "/swift-tensorflow-toolchain/usr/lib/liblldb.so"
```
A sample `launch.json` file:
```json
{
"version": "0.2.0",
"configurations": [
{
"type": "lldb",
"request": "launch",
"name": "Debug",
"program": "${workspaceFolder}/.build/x86_64-unknown-linux-gnu/debug/SwiftFusionPackageTests.xctest",
"args": [
"--enable-test-discovery"
],
"cwd": "${workspaceFolder}"
}
]
}
```
## Code overview
The main code is in `Sources/SwiftFusion`, organized under subdirectories.
### Core
The core abstractions of SwiftFusion:
- [`Vector.swift`](Sources/SwiftFusion/Core/Vector.swift): protocol that formalizes a Euclidean vector space with standard orthonormal basis, and defines a great number of default methods for it. Also defines the `ScalarsInitializableVector` and `FixedSizeVector` protocols, and a collection `StandardBasis` for the orthogonal bases.
- [`Manifold.swift`](Sources/SwiftFusion/Core/Manifold.swift): protocol that inherits from `Differentiable` that adds `retract` and `localCoordinate`, which convert between a `Manifold` structure and its `LocalCoordinate`.
- [`LieGroup.swift`](Sources/SwiftFusion/Core/LieGroup.swift): protocol that formalizes making a differentiable manifold into a Lie group with composition operator `*`. Also defines `LieGroupCoordinate` protocol.
Specific `Vector`-like data structures:
- [`VectorN.swift`](Sources/SwiftFusion/Core/VectorN.swift): generated by `VectorN.swift.gyb`, implements small fixed-size vectors conforming to `AdditiveArithmetic`, `Vector`, and `FixedSizeVector`.
- [`FixedSizeMatrix.swift`](Sources/SwiftFusion/Core/FixedSizeMatrix.swift): matrix whose dimensions are known at compile time. Conforms to `Equatable, KeyPathIterable, CustomStringConvertible, AdditiveArithmetic, Differentiable, FixedSizeVector`.
Additional utilities:
- [`DataTypes.swift`](Sources/SwiftFusion/Core/DataTypes.swift): for now, just implements `LieGroup` protocol for `Vector5`
- [`Dictionary+Differentiable.swift`](Sources/SwiftFusion/Core/Dictionary+Differentiable.swift): conform `Dictionary` to `Differentiable`.
- [`MathUtil.swift`](Sources/SwiftFusion/Core/MathUtil.swift): for now just `pinv` (pseudo-inverse), using [`Tensor.svd`](https://www.tensorflow.org/swift/api_docs/Structs/Tensor#svdcomputeuv:fullmatrices:).
- [`TensorVector.swift`](Sources/SwiftFusion/Core/TensorVector.swift): a view of a `Tensor` that conforms to the `Vector` protocol.
- [`TrappingDouble.swift`](Sources/SwiftFusion/Core/TrappingDouble.swift): a wrapper for `Double` that traps instead of allowing `NaN`.
- [`Tuple+Vector.swift`](Sources/SwiftFusion/Core/Tuple+Vector.swift): (undocumented)
- [`TypeKeyedArrayBuffers.swift`](Sources/SwiftFusion/Core/TypeKeyedArrayBuffers.swift): related to storage of `Values` and `Factors`.
### Inference
- [`FactorGraph.swift`](Sources/SwiftFusion/Inference/FactorGraph.swift): the main factor graph structure storing factors.
- [`Factor.swift`](Sources/SwiftFusion/Inference/Factor.swift): defines the core factor protocol hierarchy `Factor` → `VectorFactor` → `LinearizableFactor` → `GaussianFactor`, as well as the `VariableTuple` and `DifferentiableVariableTuple` protocols and `Tuple` conformances.
- [`GaussianFactorGraph.swift`](Sources/SwiftFusion/Inference/GaussianFactorGraph.swift): A factor graph whose factors are all `GaussianFactor`s.
A factor graph stores factors in a storage array of type `[ObjectIdentifier: AnyFactorArrayBuffer]`. [`ObjectIdentifier`](https://developer.apple.com/documentation/swift/objectidentifier) is a built-in Swift identifier type and `AnyFactorArrayBuffer` is defined in [FactorsStorage.swift](Sources/SwiftFusion/Inference/FactorsStorage.swift).
### Optimizers
- [`GradientDescent.swift`](Sources/SwiftFusion/Optimizers/GradientDescent.swift): a basic gradient descent optimizer.
- [`CGLS.swift`](Sources/SwiftFusion/Optimizers/CGLS.swift): a conjugate gradient solver for least-squares problems.
- [`LM.swift`](Sources/SwiftFusion/Optimizers/LM.swift): a Levenberg-Marquardt nonlinear optimizer.
### MCMC
- [`RandomWalkMetropolis.swift`](Sources/SwiftFusion/MCMC/RandomWalkMetropolis.swift): implements a Metropolis sampler modeled after [its equivalent](https://www.tensorflow.org/probability/api_docs/python/tfp/mcmc/RandomWalkMetropolis) in [TensorFlow Probability](https://www.tensorflow.org/probability).
### Geometry
This directory defines rotation and pose types.
We use the GTSAM naming scheme: `Pose2`, `Rot2`, `Pose3`, and `Rot3`.
- [`Pose2.swift`](Sources/SwiftFusion/Geometry/Pose2.swift): a 2D pose.
- [`Rot2.swift`](Sources/SwiftFusion/Geometry/Rot2.swift): a 2D rotation.
- [`Pose3.swift`](Sources/SwiftFusion/Geometry/Pose2.swift): a 3D pose.
- [`Rot3.swift`](Sources/SwiftFusion/Geometry/Rot3.swift): a 3D rotation.
2D and 3D points are simply represented as `Vector2` and `Vector3`.
### Image
- [`ArrayImage.swift`](Sources/SwiftFusion/Image/ArrayImage.swift): `Differentiable` multi-channel image stored as `[Double]`. Has a differentiable `tensor` method and `update` function.
- [`OrientedBoundingBox.swift`](Sources/SwiftFusion/Image/OrientedBoundingBox.swift): a rectangular region of an image, not necessarily axis-aligned.
- [`Patch.swift`](Sources/SwiftFusion/Image/Patch.swift): a differentiable `patch` method for `ArrayImage`, that returns a resampled image for the given `OrientedBoundingBox`.
### Datasets
This directory primarily contains support for reading G2O files.
# LICENSE
```
Copyright 2020 The SwiftFusion Team
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```