{"id":18810353,"url":"https://github.com/absaoss/spline-getting-started","last_synced_at":"2025-07-22T17:37:17.029Z","repository":{"id":37411902,"uuid":"316080771","full_name":"AbsaOSS/spline-getting-started","owner":"AbsaOSS","description":null,"archived":false,"fork":false,"pushed_at":"2024-09-09T12:25:58.000Z","size":15862,"stargazers_count":23,"open_issues_count":7,"forks_count":17,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-09-09T14:56:13.331Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AbsaOSS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-26T00:03:56.000Z","updated_at":"2024-09-09T12:26:03.000Z","dependencies_parsed_at":"2024-07-11T01:11:15.852Z","dependency_job_id":null,"html_url":"https://github.com/AbsaOSS/spline-getting-started","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspline-getting-started","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspline-getting-started/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspline-getting-started/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AbsaOSS%2Fspline-getting-started/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AbsaOSS","download_url":"https://codeload.github.com/AbsaOSS/spline-getting-started/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223603198,"owners_count":17172058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T23:19:55.115Z","updated_at":"2024-11-07T23:19:55.619Z","avatar_url":"https://github.com/AbsaOSS.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"Spline - data lineage tracking solution for data pipelines like Apache Spark and others\n\n---\n\n# Getting started\n\nThe project consists of three main parts:\n-  [Spark Agent](https://github.com/AbsaOSS/spline-spark-agent) that sits on a driver capturing the data lineage from Spark jobs by analyzing the execution plans\n\n-  [Rest Gateway](https://github.com/AbsaOSS/spline) that receives the lineage data from agent and stores it in the database\n\n-  [Web UI](https://github.com/AbsaOSS/spline-ui) application that visualizes the stored data lineages\n\n![Spline diagram](https://user-images.githubusercontent.com/5530211/70050339-fd93f580-15ce-11ea-88b2-4d79ee30d494.png)\n\n\n## TL;DR\nSpin up a Spline server in a Docker\n\n```shell\nwget https://raw.githubusercontent.com/AbsaOSS/spline-getting-started/main/docker/compose.yaml\n\nwget https://raw.githubusercontent.com/AbsaOSS/spline-getting-started/main/docker/.env\n\nSEED=1 docker-compose up\n# SEED=1 means to also run sample jobs to populate the database. \n```\n\nYou can access Spline services on the following URLs:\n- Spline Web UI: http://localhost:9090\n- Spline Server: http://localhost:8080\n\nTo access Spline UI from another host set `DOCKER_HOST_EXTERNAL` variable pointing to the current host before running `docker-compose`.\nSpline UI will propagate it to the user browser so that one will be able to connect to the Spline REST endpoint from the outside of this machine.\n\n```shell\nDOCKER_HOST_EXTERNAL=192.168.1.222 docker-compose up\n```\n\n## How to extend/customize _Spline Spark Agent_ behavior\n\nThere are three ways how to customize default Spline Spark Agent behavior. Choose the one that fits you needs better.\n\n1. A lot of things can be customized declaratively, without any coding needed, by just tweaking\n   the [Agent configuration](https://github.com/AbsaOSS/spline-spark-agent#configuration).\n2. Spline agent is designed for extension, so the chances are it's enough to override some method or implement some trait to achieve desired behavior,\n   and attach it as an extension module to your Spark application. See the [example extension project](spark-agent-extension-example).\n3. If the extension API isn't enough then fork the project, replace the Maven coordinates with your custom ones,\n   and [build](https://github.com/AbsaOSS/spline-spark-agent#building-for-different-scala-and-spark-versions) the agent as your own JAR.\n\n## More Howto's\n- [Running Spline on Databricks Notebook](spline-on-databricks)\n- [Setting up Spline server on AWS](spline-on-AWS-demo-setup)\n\n---\n\nFor more information about Spline see - https://absaoss.github.io/spline/\n\nEnjoy.\n\n---\n\n    Copyright 2019 ABSA Group Limited\n    \n    you may not use this file except in compliance with the License.\n    You may obtain a copy of the License at\n    \n        http://www.apache.org/licenses/LICENSE-2.0\n    \n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS,\n    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n    See the License for the specific language governing permissions and\n    limitations under the License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Fspline-getting-started","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabsaoss%2Fspline-getting-started","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabsaoss%2Fspline-getting-started/lists"}