https://github.com/kstrassheim/datawarehouse-crawler
This is a content and schema crawler tool to receive, update and import various kinds of data into a Onprem or Cloud based SQLServer or Azure-Synapse-Analysis (Azure Datawarehouse SQLServer). As source it supports SQLServer Tables, ODATA Endpoints, CSV Files or Excel Files. For multiple sources it can run in parallel mode where it would make a thread for each connection. The speciality of this crawler is that it creates the target tables by himself using the additional info from source.json. In case of Azure-Synapse-Analysis it would estimate the distribution type and keys. The syncing works completely without SQL Transactions by using a consistency correction algorithm for very frequent fact tables. There are 5 Syncing Algorithms (see Manual/Insert) which can be selected as well as one Update Algorithm.
https://github.com/kstrassheim/datawarehouse-crawler
azure-data-warehouse azure-synapse-analytics business-intelligence crawler csv data-import data-science datawarehouse datawarehousing docker dotnet-core-2 excel integration-testing odata parallel-computing sql
Last synced: 6 months ago
JSON representation
This is a content and schema crawler tool to receive, update and import various kinds of data into a Onprem or Cloud based SQLServer or Azure-Synapse-Analysis (Azure Datawarehouse SQLServer). As source it supports SQLServer Tables, ODATA Endpoints, CSV Files or Excel Files. For multiple sources it can run in parallel mode where it would make a thread for each connection. The speciality of this crawler is that it creates the target tables by himself using the additional info from source.json. In case of Azure-Synapse-Analysis it would estimate the distribution type and keys. The syncing works completely without SQL Transactions by using a consistency correction algorithm for very frequent fact tables. There are 5 Syncing Algorithms (see Manual/Insert) which can be selected as well as one Update Algorithm.
- Host: GitHub
- URL: https://github.com/kstrassheim/datawarehouse-crawler
- Owner: kstrassheim
- License: gpl-3.0
- Created: 2022-05-29T20:30:43.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-05-29T20:43:07.000Z (over 3 years ago)
- Last Synced: 2023-03-09T11:37:02.750Z (over 2 years ago)
- Topics: azure-data-warehouse, azure-synapse-analytics, business-intelligence, crawler, csv, data-import, data-science, datawarehouse, datawarehousing, docker, dotnet-core-2, excel, integration-testing, odata, parallel-computing, sql
- Language: C#
- Homepage:
- Size: 4.17 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0