{"id":13564131,"url":"https://github.com/bruin-data/ingestr","last_synced_at":"2025-05-13T15:12:04.768Z","repository":{"id":223859709,"uuid":"756594023","full_name":"bruin-data/ingestr","owner":"bruin-data","description":"ingestr is a CLI tool to copy data between any databases with a single command seamlessly.","archived":false,"fork":false,"pushed_at":"2025-05-09T15:06:34.000Z","size":175329,"stargazers_count":2951,"open_issues_count":8,"forks_count":79,"subscribers_count":20,"default_branch":"main","last_synced_at":"2025-05-09T15:27:11.828Z","etag":null,"topics":["bigquery","copy-database","data-ingestion","data-integration","data-pipeline","duckdb","ingestion-pipeline","mssql","postgresql","snowflake"],"latest_commit_sha":null,"homepage":"https://bruin-data.github.io/ingestr/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bruin-data.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"docs/supported-sources/adjust.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-12T23:00:36.000Z","updated_at":"2025-05-09T14:10:53.000Z","dependencies_parsed_at":"2024-02-22T13:25:31.699Z","dependency_job_id":"2cdc201f-14e9-49a1-9f06-690c962e420f","html_url":"https://github.com/bruin-data/ingestr","commit_stats":{"total_commits":306,"total_committers":5,"mean_commits":61.2,"dds":0.3921568627450981,"last_synced_commit":"7fd04b223a6c67ff0e460895cbf2f75970074ebb"},"previous_names":["bruin-data/ingestr"],"tags_count":123,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bruin-data%2Fingestr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bruin-data%2Fingestr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bruin-data%2Fingestr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bruin-data%2Fingestr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bruin-data","download_url":"https://codeload.github.com/bruin-data/ingestr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253962314,"owners_count":21991147,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","copy-database","data-ingestion","data-integration","data-pipeline","duckdb","ingestion-pipeline","mssql","postgresql","snowflake"],"created_at":"2024-08-01T13:01:26.967Z","updated_at":"2025-05-13T15:11:59.742Z","avatar_url":"https://github.com/bruin-data.png","language":"Python","readme":"\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/bruin-data/ingestr/blob/main/resources/ingestr.svg?raw=true\" width=\"500\" /\u003e\n    \u003cp\u003eCopy data from any source to any destination without any code\u003c/p\u003e\n    \u003cimg src=\"https://github.com/bruin-data/ingestr/blob/main/resources/demo.gif?raw=true\" width=\"750\" /\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\" style=\"margin-top: 24px;\"\u003e\n  \u003ca target=\"_blank\" href=\"https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg\" style=\"background:none\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/slack-join-dlt.svg?color=d95f5f\u0026logo=slack\" style=\"width: 180px;\"  /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n---\n\ningestr is a command-line app that allows you to ingest data from any source into any destination using simple command-line flags, no code necessary.\n\n- ✨ copy data from your database into any destination\n- ➕ incremental loading: `append`, `merge` or `delete+insert`\n- 🐍 single-command installation\n\ningestr takes away the complexity of managing any backend or writing any code for ingesting data, simply run the command and watch the data land on its destination.\n\n## Installation\nWe recommend using [uv](https://github.com/astral-sh/uv) to run `ingestr`.\n\n```\npip install uv\nuvx ingestr\n```\n\nAlternatively, if you'd like to install it globally:\n```\nuv pip install --system ingestr\n```\n\nWhile installation with vanilla `pip` is possible, it's an order of magnitude slower.\n\n## Quickstart\n\n```bash\ningestr ingest \\\n    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \\\n    --source-table 'public.some_data' \\\n    --dest-uri 'bigquery://\u003cyour-project-name\u003e?credentials_path=/path/to/service/account.json' \\\n    --dest-table 'ingestr.some_data'\n```\n\nThat's it.\n\nThis command:\n\n- gets the table `public.some_data` from the Postgres instance.\n- uploads this data to your BigQuery warehouse under the schema `ingestr` and table `some_data`.\n\n## Documentation\n\nYou can see the full documentation [here](https://bruin-data.github.io/ingestr/getting-started/quickstart.html).\n\n## Community\n\nJoin our Slack community [here](https://join.slack.com/t/bruindatacommunity/shared_invite/zt-2dl2i8foy-bVsuMUauHeN9M2laVm3ZVg).\n\n## Contributing\n\nPull requests are welcome. However, please open an issue first to discuss what you would like to change. We maybe able to offer you help and feedback regarding any changes you would like to make.\n\n\u003e [!NOTE]\n\u003e After cloning `ingestr` make sure to run `make setup` to install githooks.\n\n## Supported sources \u0026 destinations\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003cth\u003e\u003c/th\u003e\n        \u003cth\u003eSource\u003c/th\u003e\n        \u003cth\u003eDestination\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"3\" style='text-align:center;'\u003e\u003cstrong\u003eDatabases\u003c/strong\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eBigQuery\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eClickHouse\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eDatabricks\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eDuckDB\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eDynamoDB\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eLocal CSV file\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eMicrosoft SQL Server\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eMongoDB\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eMySQL\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eOracle\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003ePostgres\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n     \u003ctr\u003e\n        \u003ctd\u003eRedshift\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eSAP Hana\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eSnowflake\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eSQLite\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e❌\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"3\" style='text-align:center;'\u003e\u003cstrong\u003ePlatforms\u003c/strong\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n        \u003ctd\u003eAdjust\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eAirtable\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eAmazon Kinesis\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eApache Kafka\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eAppsFlyer\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eApp Store\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eAsana\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eChess.com\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eFacebook Ads\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eGithub\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eGorgias\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eGoogle Sheets\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eGoogle Ads\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eGoogle Analytics\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n     \u003ctr\u003e\n        \u003ctd\u003eKlaviyo\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eLinkedIn Ads\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eNotion\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003ePersonio\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n     \u003ctr\u003e\n        \u003ctd\u003eS3\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eSalesforce\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eShopify\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n     \u003ctr\u003e\n        \u003ctd\u003eSlack\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eStripe\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eTikTok Ads\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eZendesk\u003c/td\u003e\n        \u003ctd\u003e✅\u003c/td\u003e\n        \u003ctd\u003e-\u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\nFeel free to create an issue if you'd like to see support for another source or destination.\n\n## Acknowledgements\n\nThis project would not have been possible without the amazing work done by the [SQLAlchemy](https://www.sqlalchemy.org/) and [dlt](https://dlthub.com/) teams. We relied on their work to connect to various sources and destinations, and built `ingestr` as a simple, opinionated wrapper around their work.\n","funding_links":[],"categories":["Python","Table of Contents","postgresql","Data Ingestion","\u003ca name=\"Python\"\u003e\u003c/a\u003ePython","Libraries, Hooks, Utilities"],"sub_categories":["Data Pipeline","Libraries"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbruin-data%2Fingestr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbruin-data%2Fingestr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbruin-data%2Fingestr/lists"}