Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/roeap/flight-fusion
https://github.com/roeap/flight-fusion
arrow data-science datafusion deltalake flight
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/roeap/flight-fusion
- Owner: roeap
- License: apache-2.0
- Created: 2021-11-23T09:06:27.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-06-22T04:38:59.000Z (over 1 year ago)
- Last Synced: 2023-06-25T23:06:56.737Z (over 1 year ago)
- Topics: arrow, data-science, datafusion, deltalake, flight
- Language: Rust
- Homepage:
- Size: 3.96 MB
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 29
-
Metadata Files:
- Readme: README.md
- License: LICENSE-APACHE
Awesome Lists containing this project
README
# flight-fusion
This repository contains some experiments on what a modern data platform could look like.
A strong emphasis lies on how observability can be achieved throughout all actions
occurring on a data platform.
## Documentation
Currently the documentation is not published, but can be viewed in this repository.
```sh
mkdocs serve
```
## Development
### Prerequisites
- rust 1.58+
- python 3.8+
- poetry (for python development)
- [just](https://github.com/casey/just) (for running workspace commands)
### Rust
To run unit tests for the rust crates execute
```sh
cargo test
```
To build crates and services
```sh
cargo build
```
TODO explain mock framework
## Python Development
To set up the python environment run
```sh
poetry install
```
The python bindings for rust dependent crates have to be build and installed separately.
Make sure you have the environment created by poetry activated and run.
```sh
make python-develop
```
## Dagster
To see the Dagster integrations in play, create two folders in the project root: `.dagster` and `.fusion`,
then start an instance of the fusion service in a separate shell.
```sh
fusion server start --host 127.0.0.1 --port 50051 --log-level info
```
An example Dagster repository using most of the features from the `dagster-fusion` package is
provided within `examples/model-training`. To inspect it and play with the configurations,
run a local instance of dagster.
## Build Docker
```sh
docker build -f docker/Dockerfile
```
```sh
docker run -d -p6831:6831/udp -p6832:6832/udp -p16686:16686 jaegertracing/all-in-one:latest
```
### Generating python protos
To generate proto definitions for use in python client.
```
make python-proto
```
#### License
Licensed under either of Apache License, Version
2.0 or MIT license at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in this crate by you, as defined in the Apache-2.0 license, shall
be dual licensed as above, without any additional terms or conditions.