{"id":18810372,"url":"https://github.com/absaoss/spline-openlineage","last_synced_at":"2025-04-13T20:30:59.939Z","repository":{"id":39996570,"uuid":"480820334","full_name":"AbsaOSS/spline-openlineage","owner":"AbsaOSS","description":"Open Lineage Integration for Spline","archived":false,"fork":false,"pushed_at":"2022-09-25T13:14:34.000Z","size":142,"stargazers_count":5,"open_issues_count":2,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-04-24T12:06:58.485Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AbsaOSS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-12T13:22:35.000Z","updated_at":"2023-11-18T09:56:39.000Z","dependencies_parsed_at":"2022-08-31T07:23:02.664Z","dependency_job_id":null,"html_url":"https://github.com/AbsaOSS/spline-openlineage","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspline-openlineage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspline-openlineage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspline-openlineage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspline-openlineage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AbsaOSS","download_url":"https://codeload.github.com/AbsaOSS/spline-openlineage/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223603260,"owners_count":17172071,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T23:20:00.625Z","updated_at":"2024-11-07T23:20:01.235Z","avatar_url":"https://github.com/AbsaOSS.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spline OpenLineage Integration\nApplications in this repo provide a way to send an **open lineage** messages into **spline server**.\n\n\n- **rest-proxy** provides rest open lineage api and inserts all messages directly to kafka\n- **aggregator** consumes the messages form kafka and merges all messages for one run. \nOnce the `COMPLETE` message arrives all the data are converted into one Spline Plan and one Event message. \nIn the last step messages are inserted into another kafka topic where they can be consumed by spline kafka server.\n\n### Architecture\n\n![](./documentation/img/spline-ol-integration.svg)\n\n### Architecture with rest of Spline components\n\n![](./documentation/img/spline-ol-integration-full.svg)\n\n### Hwo To Build\nBuilding artifacts jar/war\n```bash\nmvn clean package\n```\n\nBuilding docker images\n```bash\nmvn clean install -Pdocker -Ddockerfile.repositoryUrl=my\n```\n\n### Docker Compose\nThere are two docker-compose files to make it easy to experiment with the project:\nThe one in `docker/ol-integration/` will create environment and start artifacts from this repo only.\nThe other one `docker/ol-integration-and-spline` will start the whole Spline pipeline from open lineage message ingestion to spline ui at the end.\n\n### How To Run\n#### Prerequisities\n- Running kafka broker, for this example it is on `localhost:9092`\n- Two kafka topics (one for messages in open lineage format and one for messages in spline format). \nFor this example it will be `open-lineage-topic` and `spline-lineage-topic`\n\n#### Runing in docker\nFor rest endpoint I have to map the port so I am able to acces it outside of docker and also set the open lineage topic and addres of kafka broker. \nIt's important to replace localhost for appropriate address that works from inside the container. For macOS it is `host.docker.internal`.\n\n```\ndocker run -p 8086:8080 \\\n    -e JAVA_OPTS=\"-Dspline.ol.topic=open-lineage-topic -Dspline.ol.producer.bootstrap.servers=host.docker.internal:9092\" \\\n    my/spline-open-lineage-rest-proxy\n```\n\n```\ndocker run \\\n    -e JAVA_OPTS=\"-Dspline.ol.inputTopic=open-lineage-topic -Dspline.ol.outputTopic=spline-lineage-topic -Dspline.ol.streams.bootstrap.servers=host.docker.internal:9092\" \\\n    my/spline-open-lineage-aggregator\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Fspline-openlineage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabsaoss%2Fspline-openlineage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Fspline-openlineage/lists"}