https://github.com/bruin-data/ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
https://github.com/bruin-data/ingestr

bigquery copy-database data-ingestion data-integration data-pipeline duckdb ingestion-pipeline mssql postgresql snowflake

Last synced: 7 days ago
JSON representation

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

• Host: GitHub
• URL: https://github.com/bruin-data/ingestr
• Owner: bruin-data
• License: mit
• Created: 2024-02-12T23:00:36.000Z (11 months ago)
• Default Branch: main
• Last Pushed: 2024-12-12T19:46:30.000Z (18 days ago)
• Last Synced: 2024-12-12T20:37:53.850Z (18 days ago)
• Topics: bigquery, copy-database, data-ingestion, data-integration, data-pipeline, duckdb, ingestion-pipeline, mssql, postgresql, snowflake
• Language: Python
• Homepage: https://bruin-data.github.io/ingestr/
• Size: 163 MB
• Stars: 2,582
• Watchers: 18
• Forks: 65
• Open Issues: 15
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md
  • Support: docs/supported-sources/adjust.md

Awesome Lists containing this project

• awesome-github-repos - bruin-data/ingestr - ingestr is a CLI tool to copy data between any databases with a single command seamlessly. (Python)
• awesome-list - bruin-data/ingestr - ingestr is a CLI tool to copy data between any databases with a single command seamlessly. (Python)

README

        


    

    
Copy data from any source to any destination without any code

    





  

    

  



---

ingestr is a command-line app that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.

- ✨ copy data from your database into any destination

- ➕ incremental loading: `append`, `merge` or `delete+insert`

- 🐍 single-command installation

ingestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.

## Installation

We recommend using [uv](https://github.com/astral-sh/uv) to run `ingestr`.

```

pip install uv

uvx ingestr

```

Alternatively, if you'd like to install it globally:

```

uv pip install --system ingestr

```

While installation with vanilla `pip` is possible, it's an order of magnitude slower.

## Quickstart

```bash

ingestr ingest \

    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \

    --source-table 'public.some_data' \

    --dest-uri 'bigquery://?credentials_path=/path/to/service/account.json' \

    --dest-table 'ingestr.some_data'

```

That's it.

This command:

- gets the table `public.some_data` from the Postgres instance.

- uploads this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.

## Documentation

You can see the full documentation [here](https://bruin-data.github.io/ingestr/getting-started/quickstart.html).

## Community

Join our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).

## Contributing

Pull requests are welcome. However, please open an issue first to discuss what you would like to change. We maybe able to offer you help and feedback regarding any changes you would like to make.

> [!NOTE]

> After cloning `ingestr` make sure to run `make setup` to install githooks.

## Supported sources & destinations

    

        

        Source

        Destination

    

    

        Databases

    

    

        Postgres

        ✅

        ✅

    

    

        BigQuery

        ✅

        ✅

    

    

        Snowflake

        ✅

        ✅

    

    

        Redshift

        ✅

        ✅

    

    

        Databricks

        ✅

        ✅

    

    

        DuckDB

        ✅

        ✅

    

    

        Microsoft SQL Server

        ✅

        ✅

    

    

        Local CSV file

        ✅

        ✅

    

    

        MongoDB

        ✅

        ❌

    

    

        Oracle

        ✅

        ❌

    

    

        SAP Hana

        ✅

        ❌

    

    

        SQLite

        ✅

        ❌

    

    

        MySQL

        ✅

        ❌

    

    

        Platforms

    

        Adjust

        ✅

        -

    

        Airtable

        ✅

        -

    

    

        AppsFlyer

        ✅

        -

    

    

        Asana

        ✅

        -

    

    

        Chess.com

        ✅

        -

    

    

        DynamoDB

        ✅

        -

    

    

        Facebook Ads

        ✅

        -

    

    

        Gorgias

        ✅

        -

    

    

        Google Sheets

        ✅

        -

    

    

        HubSpot

        ✅

        -

    

     

        Klaviyo

        ✅

        -

    

    

        Notion

        ✅

        -

    

     

        S3

        ✅

        -

    

    

        Shopify

        ✅

        -

    

     

        Slack

        ✅

        -

    

    

        Stripe

        ✅

        -

    

    

        Zendesk

        ✅

        -

    

Feel free to create an issue if you'd like to see support for another source or destination.

## Acknowledgements

This project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.