{"id":17061391,"url":"https://github.com/roeap/flight-fusion","last_synced_at":"2025-04-12T18:12:42.382Z","repository":{"id":37086488,"uuid":"431030777","full_name":"roeap/flight-fusion","owner":"roeap","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-16T21:34:32.000Z","size":4514,"stargazers_count":7,"open_issues_count":27,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-12T18:12:37.384Z","etag":null,"topics":["arrow","data-science","datafusion","deltalake","flight"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/roeap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-23T09:06:27.000Z","updated_at":"2025-03-12T19:24:33.000Z","dependencies_parsed_at":"2023-02-16T18:15:41.516Z","dependency_job_id":null,"html_url":"https://github.com/roeap/flight-fusion","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roeap%2Fflight-fusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roeap%2Fflight-fusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roeap%2Fflight-fusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roeap%2Fflight-fusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/roeap","download_url":"https://codeload.github.com/roeap/flight-fusion/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248610341,"owners_count":21132919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arrow","data-science","datafusion","deltalake","flight"],"created_at":"2024-10-14T10:46:58.662Z","updated_at":"2025-04-12T18:12:42.360Z","avatar_url":"https://github.com/roeap.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# flight-fusion\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://github.com/roeap/flight-fusion/actions/workflows/python.yml\"\u003e\u003cimg alt=\"Actions Status\" src=\"https://github.com/roeap/flight-fusion/actions/workflows/python.yml/badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/roeap/flight-fusion/actions/workflows/rust.yml\"\u003e\u003cimg alt=\"Actions Status\" src=\"https://github.com/roeap/flight-fusion/actions/workflows/rust.yml/badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/roeap/flight-fusion/actions/workflows/rust.yml\"\u003e\u003cimg alt=\"Rustc Version 1.62+\" src=\"https://img.shields.io/badge/rustc-1.58+-lightgray.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://codecov.io/gh/roeap/flight-fusion\"\u003e\u003cimg src=\"https://codecov.io/gh/roeap/flight-fusion/branch/main/graph/badge.svg?token=QI8UWIJ8KY\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nThis repository contains some experiments on what a modern data platform could look like.\nA strong emphasis lies on how observability can be achieved throughout all actions\noccurring on a data platform.\n\n## Documentation\n\nCurrently the documentation is not published, but can be viewed in this repository.\n\n```sh\nmkdocs serve\n```\n\n## Development\n\n### Prerequisites\n\n- rust 1.58+\n- python 3.8+\n- poetry (for python development)\n- [just](https://github.com/casey/just) (for running workspace commands)\n\n### Rust\n\nTo run unit tests for the rust crates execute\n\n```sh\ncargo test\n```\n\nTo build crates and services\n\n```sh\ncargo build\n```\n\nTODO explain mock framework\n\n## Python Development\n\nTo set up the python environment run\n\n```sh\npoetry install\n```\n\nThe python bindings for rust dependent crates have to be build and installed separately.\nMake sure you have the environment created by poetry activated and run.\n\n```sh\nmake python-develop\n```\n\n## Dagster\n\nTo see the Dagster integrations in play, create two folders in the project root: `.dagster` and `.fusion`,\nthen start an instance of the fusion service in a separate shell.\n\n```sh\nfusion server start --host 127.0.0.1 --port 50051 --log-level info\n```\n\nAn example Dagster repository using most of the features from the `dagster-fusion` package is\nprovided within `examples/model-training`. To inspect it and play with the configurations,\nrun a local instance of dagster.\n\n## Build Docker\n\n```sh\ndocker build -f docker/Dockerfile\n```\n\n```sh\ndocker run -d -p6831:6831/udp -p6832:6832/udp -p16686:16686 jaegertracing/all-in-one:latest\n```\n\n### Generating python protos\n\nTo generate proto definitions for use in python client.\n\n```\nmake python-proto\n```\n\n#### License\n\n\u003csup\u003e\nLicensed under either of \u003ca href=\"LICENSE-APACHE\"\u003eApache License, Version\n2.0\u003c/a\u003e or \u003ca href=\"LICENSE-MIT\"\u003eMIT license\u003c/a\u003e at your option.\n\u003c/sup\u003e\n\n\u003cbr\u003e\n\n\u003csub\u003e\nUnless you explicitly state otherwise, any contribution intentionally submitted\nfor inclusion in this crate by you, as defined in the Apache-2.0 license, shall\nbe dual licensed as above, without any additional terms or conditions.\n\u003c/sub\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froeap%2Fflight-fusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froeap%2Fflight-fusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froeap%2Fflight-fusion/lists"}