{"id":39698198,"url":"https://github.com/databricks-solutions/lakefed-ingest","last_synced_at":"2026-01-18T10:19:08.673Z","repository":{"id":320032136,"uuid":"1067504890","full_name":"databricks-solutions/lakefed-ingest","owner":"databricks-solutions","description":"Ingest data into Databricks with Lakehouse Federation","archived":false,"fork":false,"pushed_at":"2026-01-13T03:37:08.000Z","size":1191,"stargazers_count":1,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-13T06:48:51.137Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks-solutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS.txt","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE.md","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-01T00:42:08.000Z","updated_at":"2025-12-07T21:08:56.000Z","dependencies_parsed_at":"2025-10-21T16:25:12.710Z","dependency_job_id":null,"html_url":"https://github.com/databricks-solutions/lakefed-ingest","commit_stats":null,"previous_names":["databricks-solutions/lakefed-ingest"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/databricks-solutions/lakefed-ingest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Flakefed-ingest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Flakefed-ingest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Flakefed-ingest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Flakefed-ingest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks-solutions","download_url":"https://codeload.github.com/databricks-solutions/lakefed-ingest/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Flakefed-ingest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28534316,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T10:13:46.436Z","status":"ssl_error","status_checked_at":"2026-01-18T10:13:11.045Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-18T10:19:07.949Z","updated_at":"2026-01-18T10:19:08.664Z","avatar_url":"https://github.com/databricks-solutions.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lakehouse Federation Ingestion Framework\n\nMetadata-driven framework for ingesting data into Databricks using [Lakehouse Federation](https://docs.databricks.com/en/query-federation/index.html). Supports the following ingestion patterns:\n- **Full**: ingests entire table\n- **Incremental**: ingests incrementally using watermarks\n- **Partitioned**: spreads ingestion across many small queries, run N at a time. Used for large tables. See diagram below.\n\n![Lakehouse Federation ingest diagram](assets/lakefed_ingest_diagram.png \"Lakehouse Federation ingest diagram\")\n\nThe following sources are currently supported:\n- SQL Server\n- Oracle\n- PostgreSQL\n- Redshift\n- Synapse\n\n## Prerequisites\n\n### 1. Setup Lakehouse Federation\nFollow the [Lakehouse Federation](https://docs.databricks.com/en/query-federation/index.html) instructions to create a connection and foreign catalog\n\n### 2. Install the Databricks CLI\nInstall the Databricks CLI from https://docs.databricks.com/dev-tools/cli/install.html\n\n### 3. Authenticate to your Databricks workspace\nChoose one of the following authentication methods:\n\n#### Option A: Personal Access Token (PAT)\n\n1. **Generate Personal Access Token:**\n   - Log into your Databricks workspace\n   - Click on your username in the top-right corner\n   - Select **User Settings** → **Developer** → **Access tokens**\n   - Click **Generate new token**\n   - Give it a name (e.g., \"Local Development\") and set expiration\n   - Copy the generated token\n\n2. **Configure CLI with PAT:**\n   ```bash\n   databricks configure --token --profile DEFAULT\n   ```\n   \n   You'll be prompted for:\n   - **Databricks Host**: `https://your-workspace.cloud.databricks.com`\n   - **Token**: Paste your generated token\n\n    This will update DEFAULT profile in `~/.databrickscfg` \n\n#### Option B: OAuth Authentication\n\nConfigure OAuth:\n\n```bash\ndatabricks auth login --host https://your-workspace.cloud.databricks.com --profile PROD\n```\n\nThis will:\n- Open your browser for authentication\n- Create a profile in `~/.databrickscfg`\n- Store OAuth credentials securely\n\n#### Verify Configuration\n\nCheck your configuration:\n\n```bash\n# List all profiles\ncat ~/.databrickscfg\n```\n\nYour `~/.databrickscfg` should look like:\n\n```ini\n[DEFAULT]\nhost = https://your-workspace.cloud.databricks.com\ntoken = dapi123abc...\n\n[DEV]\nhost = https://dev-workspace.cloud.databricks.com\ntoken = dapi456def...\n\n[PROD]\nhost = https://prod-workspace.cloud.databricks.com\ntoken = databricks-cli\n```\n\n### 4. Set up Python Virtual Environment\nCreate and activate a [Python virtual environment](https://realpython.com/python-virtual-environments-a-primer/) to manage dependencies:\n\n```bash\n# Create virtual environment on macOS/Linux\n# See link above for Windows documentation\n$ python3 -m venv .venv\n\n# Activate virtual environment\n$ source .venv/bin/activate\n\n# Install required Python packages\n$ pip install -r requirements-dev.txt\n```\n\n### 5. Configure databricks.yml Variables\nUpdate the variables in `databricks.yml` to match your environment.\n\n- **workspace.host**: Your Databricks workspace URL\n- **cluster_id**: ID of your cluster for production deployment. For development, the bundle will lookup the ID based on the specified name (Eg, Shared Cluster).\n- **warehouse_id**: ID of your SQL warehouse for production deployment. For development, the bundle will lookup the ID based on the specified name (Eg, Shared Serverless).\n- **concurrency**: Concurrency of for each tasks. Can be overridden during deployment.\n\nExample configuration for dev target:\n```yaml\ntargets:\n  dev:\n    mode: development\n    default: true\n    workspace:\n      host: https://your-workspace.cloud.databricks.com\n    variables:\n      cluster_id: your_cluster_id\n      warehouse_id: your_warehouse_id\n      concurrency: 16\n```\n\n## Getting Started\n\n### 1. Add Metadata to Control Table\nThe solution is driven by metadata stored in a control table. In this table you can specific sources and sinks, loading behavior (Full, incremental, partitioned), etc.\n1. Create the control table using the [_create_control_table](notebooks/_create_control_table.ipynb) notebook.\n2. Merge metadata into the control table. See the [load_metadata_tpcds](notebooks/load_metadata_tpcds.ipynb) notebook for an example.\n\n### 2. Configure Sources\nSome sources require additional configuration in order to retrieve table sizes for partitioned ingestion:\n\n**Oracle**  \nIngesting from Oracle requires permission to read the sys.dba_segments table. This is to obtain the source table size.\n\n**PostgreSQL**  \nThe number of queries used for ingestion is determined in part by the size of the source table. Since Lakehouse Federation doesn't currently support PostgreSQL object size functions (E.g., pg_table_size), you need to create a view in the source database or use JDBC pushdown. **Creating a view in the source database is strongly recommended.**\n\n1. Database view - create a view in the source database using the statement below. Leave the `jdbc_config_file` job parameter blank, and the view will be queried using Lakehouse Federation.\n\n```sql\ncreate or replace view public.vw_pg_table_size\n as\n select\n  table_schema,\n  table_name,\n  pg_table_size(quote_ident(table_name)),\n  pg_size_pretty(pg_table_size(quote_ident(table_name))) as pg_table_size_pretty\nfrom information_schema.tables\nwhere table_schema not in ('pg_catalog', 'information_schema')\nand table_type = 'BASE TABLE';\n```\n\n2. JDBC pushdown - create a config file like [config/postgresql_jdbc.json](config/postgresql_jdbc.json). Use the path to the file as the value for the `jdbc_config_file` job parameter. [Secrets](https://docs.databricks.com/aws/en/security/secrets/) must be used for JDBC credentials. See [notebooks/manage_secrets.ipynb](notebooks/manage_secrets.ipynb) for reference.\n\n### 3. Run Controller Job\n1. Run the lakefed_ingest_controller job, providing the desired task_collection as a parameter.\n2. The lakefed_ingest_controller job will run all non-partitioned tasks, followed by all partitioned tasks. Non-partitioned tasks run concurrently, and partitioned tasks run sequentially. This is because partitioned tasks will spawn concurrent queries, and we want to maintain a consistent level of concurrency at the controller job (And source system) scope.\n\n### Recommendations\n- Use a partition column with a relatively even distribution. If the partition column is also used in an index, that is even better.\n- Use a small all-purpose cluster if you have partitioned ingestion tasks. This cluster is used only for configuring partitions (Not heavy data processing), and we don't want to wait for a job cluster to spin up for each partitioned ingestion task.\n\n### Limitations\n- Does not handle skew. The solution works best when the partition column has an even distribution.\n- Does not provide atomicity. Individual queries are not executed as a single transaction. One could fail while the rest succeed, or the source table could be altered before all ingestion queries are completed.\n\n## Deployment\n\n### Deploy to Development Environment\n```bash\n$ databricks bundle deploy --target dev --profile DEFAULT\n```\nNote: Since \"dev\" is specified as the default target in databricks.yml, you can omit the `--target dev` parameter. Similarly, `--profile DEFAULT` can be omitted if you only have one profile configured for your workspace.\n\nThis deploys everything that's defined for this project, including:\n- Three jobs prefixed with `lakefed_ingest_`\n- main.py module for the partitioned ingest job\n- All associated resources\n\nYou can find the deployed job by opening your workspace and clicking on **Workflows**.\n\n### Deploy to Production Environment\n```bash\n$ databricks bundle deploy --target prod --profile PROD\n```\n\n### Run a Job\n```bash\n$ databricks bundle run --target prod --profile PROD\n```\n\n## Development Tools\n\nFor enhanced development experience, consider installing:\n- Databricks extension for Visual Studio Code: https://docs.databricks.com/dev-tools/vscode-ext.html\n\n## Documentation\n\nFor comprehensive documentation on:\n- **Databricks Asset Bundles**: https://docs.databricks.com/dev-tools/bundles/index.html\n- **CI/CD configuration**: https://docs.databricks.com/dev-tools/bundles/index.html\n\n## Project Structure\n\n- `assets/`: Images for README\n- `config/`: Config for PostgreSQL JDBC pushdown\n- `notebooks/`: Notebooks showing how to load metadata and work with Databricks Secrets\n- `resources/`: Databricks Asset Bundle resource definitions\n- `src/`: Source files including notebooks, SQL files, and Python modules\n- `databricks.yml`: Main bundle configuration file\n\n## Contributing\n\n### 1. Create and Activate Virtual Environment\nFollow the instructions above in the \"Set up Python Virtual Environment\" section.\n\n### 2. Run Unit Tests\n\nDatabricks Connect is required to run some of the unit tests. \n\n1. Install dependent packages:\n```\n$ pip install -r requirements-dev.txt\n```\n\n2. Run unit tests with pytest\n```\n$ pytest\n```\n\nIf you run into this error:\n```\nERROR tests/main_test.py - Exception: Cluster id or serverless are required but were not specified.\n```\n\nAdd the cluster_id to your .databrickscfg file\n```\n[DEFAULT]\nhost = https://your-workspace.cloud.databricks.com\ncluster_id = XXXX-XXXXXX-XXXXXXXX\nauth_type  = databricks-cli\n```\n\n## How to get help\nDatabricks support doesn't cover this content. For questions or bugs, please open a GitHub issue and the team will help on a best effort basis.\n\n## License\n\u0026copy; 2025 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the [Databricks License](https://databricks.com/db-license-source). All included or referenced third party libraries are subject to the licenses set forth below.\n\n| library | description | license | source |\n|---------|-----------|-----------|------|\n| pytest | Testing framework | MIT | [GitHub](https://github.com/pytest-dev/pytest) |\n| setuptools | Build system | MIT | [GitHub](https://github.com/pypa/setuptools) |\n| wheel | CLI for manipulating wheel files | MIT | [GitHub](https://github.com/pypa/wheel) |\n| jsonschema | JSON schema validation | MIT | [GitHub](https://github.com/python-jsonschema/jsonschema) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-solutions%2Flakefed-ingest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks-solutions%2Flakefed-ingest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-solutions%2Flakefed-ingest/lists"}