Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bruin-data/ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
https://github.com/bruin-data/ingestr
bigquery copy-database data-ingestion data-integration data-pipeline duckdb ingestion-pipeline mssql postgresql snowflake
Last synced: about 2 months ago
JSON representation
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
- Host: GitHub
- URL: https://github.com/bruin-data/ingestr
- Owner: bruin-data
- License: mit
- Created: 2024-02-12T23:00:36.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-04-24T22:09:36.000Z (5 months ago)
- Last Synced: 2024-04-25T00:03:25.437Z (5 months ago)
- Topics: bigquery, copy-database, data-ingestion, data-integration, data-pipeline, duckdb, ingestion-pipeline, mssql, postgresql, snowflake
- Language: Python
- Homepage: https://bruin-data.github.io/ingestr/
- Size: 721 KB
- Stars: 2,315
- Watchers: 14
- Forks: 47
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
- Support: docs/supported-sources/bigquery.md
Awesome Lists containing this project
- awesome-github-repos - bruin-data/ingestr - ingestr is a CLI tool to copy data between any databases with a single command seamlessly. (Python)
- awesome-list - bruin-data/ingestr - ingestr is a CLI tool to copy data between any databases with a single command seamlessly. (Python)
README
Ingest & copy data from any source to any destination without any code
-----
Ingestr is a command-line application that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.
- ✨ copy data from your database into any destination
- ➕ incremental loading: `append`, `merge` or `delete+insert`
- 🐍 single-command installation
ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.
## Installation
```
pip install ingestr
```
## Quickstart
```bash
ingestr ingest \
--source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
--source-table 'public.some_data' \
--dest-uri 'bigquery://?credentials_path=/path/to/service/account.json' \
--dest-table 'ingestr.some_data'
```
That's it.
This command will:
- get the table `public.some_data` from the Postgres instance.
- upload this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.
## Documentation
You can see the full documentation [here](https://bruin-data.github.io/ingestr/getting-started/quickstart.html).
## Community
Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).
## Supported Sources & Destinations
Source
Destination
Databases
Postgres
✅
✅
BigQuery
✅
✅
Snowflake
✅
✅
Redshift
✅
✅
Databricks
✅
✅
DuckDB
✅
✅
Microsoft SQL Server
✅
✅
Local CSV file
✅
✅
MongoDB
✅
❌
Oracle
✅
❌
SAP Hana
✅
❌
SQLite
✅
❌
MySQL
✅
❌
Platforms
Gorgias
✅
❌
Google Sheets
✅
❌
Notion
✅
❌
Shopify
✅
❌
More to come soon!
## Acknowledgements
This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.