{"id":15021496,"url":"https://github.com/smart-data-lake/smart-data-lake","last_synced_at":"2025-04-13T05:07:41.091Z","repository":{"id":36966690,"uuid":"201033534","full_name":"smart-data-lake/smart-data-lake","owner":"smart-data-lake","description":"Smart Automation Tool for building modern Data Lakes and Data Pipelines","archived":false,"fork":false,"pushed_at":"2025-04-06T14:09:12.000Z","size":44949,"stargazers_count":121,"open_issues_count":58,"forks_count":22,"subscribers_count":14,"default_branch":"develop-spark3","last_synced_at":"2025-04-13T05:07:35.009Z","etag":null,"topics":["data-lake","data-pipelines","deltalake","hadoop","hive","scala","smart-data-lake","spark","transform-data"],"latest_commit_sha":null,"homepage":"https://www.smartdatalake.io","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smart-data-lake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":"GOVERNANCE.md","roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-07T11:07:24.000Z","updated_at":"2025-04-01T13:40:09.000Z","dependencies_parsed_at":"2023-10-23T20:32:35.974Z","dependency_job_id":"e8a77580-81cb-4df6-8669-31ce95ede262","html_url":"https://github.com/smart-data-lake/smart-data-lake","commit_stats":{"total_commits":1534,"total_committers":41,"mean_commits":37.41463414634146,"dds":0.6297262059973925,"last_synced_commit":"96b5c8ef25ea395245b1d1a97c5701b14598818d"},"previous_names":[],"tags_count":42,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smart-data-lake%2Fsmart-data-lake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smart-data-lake%2Fsmart-data-lake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smart-data-lake%2Fsmart-data-lake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smart-data-lake%2Fsmart-data-lake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smart-data-lake","download_url":"https://codeload.github.com/smart-data-lake/smart-data-lake/tar.gz/refs/heads/develop-spark3","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248665747,"owners_count":21142123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-lake","data-pipelines","deltalake","hadoop","hive","scala","smart-data-lake","spark","transform-data"],"created_at":"2024-09-24T19:56:38.540Z","updated_at":"2025-04-13T05:07:41.071Z","avatar_url":"https://github.com/smart-data-lake.png","language":"Scala","funding_links":[],"categories":["Tools","Table of Contents","大数据"],"sub_categories":["Lakehouse System","Spring Cloud框架"],"readme":"# Smart Data Lake\n![Build Status](https://github.com/smart-data-lake/smart-data-lake/actions/workflows/snapshot_builds.yml/badge.svg)\n \nSmart Data Lake Builder is a data lake automation framework that makes loading and transforming data a breeze. \nIt is implemented in Scala and builds on top of open-source big data technologies like [Apache Hadoop](https://hadoop.apache.org/) and [Apache Spark](https://spark.apache.org/), including connectors for diverse data sources (HadoopFS, Hive, DeltaLake, JDBC, Splunk,  Webservice, SFTP, JMS, Excel, Access) and file formats.\n\n### A Data Lake\n* is a central raw data store for analytics\n* facilitates cheap raw storage to handle growing volumes of data\n* enables topnotch artificial intelligence (AI) and machine learning (ML) technologies for data-driven enterprises\n\n### The Smart Data Lake adds\n* a layered data architecture to provide not only raw data, but prepared, secured, high quality data according to business entities, ready to use for analytical use cases, also called «Smart Data». This is comparable to Databricks Lake House architecture, in fact Smart Data Lake Builder is a very good choice to automate a Lake House, also on Databricks.\n* a declarative, configuration-driven approach to creating data pipelines. Metadata about data pipelines allows for efficient operations, maintenance and more business self-service.\n\n### Benefits of Smart Data Lake Builder\n* Cost-effective implementation of data lakes\n* Increased productivity of data scientists\n* Higher level of self-service\n* Decreased operations and maintenance costs\n* Fully open source, no vendor lock-in\n\n### When should you consider using Smart Data Lake Builder ?\nSome common use cases include:\n* Building Data Lakes, drastically increasing productivity and usability \n* Data Apps - building complex data processing apps\n* DWH automation - reading and writing to relational databases via SQL \n* Data migration - Efficiently create one-time data pipelines\n* Data Catalog / Data Lineage - Generated automatically from metadata \n\nSee [Features](https://github.com/smart-data-lake/smart-data-lake/blob/documentation/docs/features.md) for a comprehensive list of Smart Data Lake Builder features.\n\n## How it works\nThe following diagram shows the core concepts:\n\n![How it works](https://github.com/smart-data-lake/smart-data-lake/blob/documentation/docs/images/feed.png)\n\n### Data object\nA data object defines the location and format of data. \nSome data objects require a connection to access remote data (e.g. a database connection).\n\n### Action\nThe \"data processors\" are called actions. \nAn action requires at least one input and output data object. \nAn action reads the data from the input data object, processes and writes it to the output data object. \nMany actions are predefined e.g. transform data from json to csv but you can also define your custom transformer action.\n\n### Feed\nActions connect different Data Object and implicitly define a directed acyclic graph, as they model the dependencies needed to fill a Data Object.\nThis automatically generated, arbitrary complex data flow can be divided up into Feed's (subgraphs) for execution and monitoring.\n\n### Configuration\nAll metadata i.e. connections, data objects and actions are defined in a central configuration file, usually called application.conf.\nThe file format used is [HOCON](https://github.com/lightbend/config/blob/master/HOCON.md) which makes it easy to edit.\n\n### Getting Started\nTo see how all this works in action, head over to the [Getting Started](https://smartdatalake.ch/docs/getting-started/setup) page.\n\n# Major Contributors\n![SBB](https://github.com/smart-data-lake/smart-data-lake/blob/documentation/docs/images/SBB_logo.png)  \n[www.sbb.ch](http://www.sbb.ch) : Provided the previously developed software as a foundation for the open source project   \n\n![ELCA](https://github.com/smart-data-lake/smart-data-lake/blob/documentation/docs/images/ELCA_logo.png)  \n[www.elca.ch](http://www.elca.ch) : Did the comprehensive revision and provision as open source project \n\n# Additional Documentation\n[Getting Started](https://smartdatalake.ch/docs/getting-started/setup)  \n[Reference](https://smartdatalake.ch/docs/reference/build)  \n[Architecture](https://smartdatalake.ch/docs/architecture)  \n[Testing](https://github.com/smart-data-lake/smart-data-lake/blob/documentation/docs/reference/testing.md)  \n[Glossary](https://github.com/smart-data-lake/smart-data-lake/blob/documentation/docs/reference/glossary.md)  \n[Troubleshooting](https://smartdatalake.ch/docs/getting-started/troubleshooting/common-problems)    \n[FAQ](docs/FAQ.md)  \n[Contributing](CONTRIBUTING.MD)  \n[Running in the Public Cloud](docs/PublicCloud.md)  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmart-data-lake%2Fsmart-data-lake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmart-data-lake%2Fsmart-data-lake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmart-data-lake%2Fsmart-data-lake/lists"}