{"id":13459577,"url":"https://github.com/getindata/kedro-snowflake","last_synced_at":"2025-04-09T20:11:36.321Z","repository":{"id":155579305,"uuid":"570499128","full_name":"getindata/kedro-snowflake","owner":"getindata","description":"Kedro Snowflake / Snowpark plugin","archived":false,"fork":false,"pushed_at":"2024-07-19T08:53:25.000Z","size":5893,"stargazers_count":13,"open_issues_count":1,"forks_count":2,"subscribers_count":10,"default_branch":"develop","last_synced_at":"2025-04-09T20:11:30.127Z","etag":null,"topics":["kedro","machine-learning","mlops","snowflake","snowpark"],"latest_commit_sha":null,"homepage":"https://kedro-snowflake.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/getindata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-25T10:39:58.000Z","updated_at":"2024-11-21T09:26:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"929e0df7-3e17-413c-8f3a-4c0c4f768928","html_url":"https://github.com/getindata/kedro-snowflake","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fkedro-snowflake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fkedro-snowflake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fkedro-snowflake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fkedro-snowflake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/getindata","download_url":"https://codeload.github.com/getindata/kedro-snowflake/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248103872,"owners_count":21048245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kedro","machine-learning","mlops","snowflake","snowpark"],"created_at":"2024-07-31T10:00:19.990Z","updated_at":"2025-04-09T20:11:36.291Z","avatar_url":"https://github.com/getindata.png","language":"Python","funding_links":[],"categories":["[Kedro plugins](https://docs.kedro.org/en/stable/extend_kedro/plugins.html)"],"sub_categories":[],"readme":"# Kedro Snowflake Pipelines plugin\n\n[![Python Version](https://img.shields.io/pypi/pyversions/kedro-snowflake)](https://github.com/getindata/kedro-snowflake)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![SemVer](https://img.shields.io/badge/semver-2.0.0-green)](https://semver.org/)\n[![PyPI version](https://badge.fury.io/py/kedro-snowflake.svg)](https://pypi.org/project/kedro-snowflake/)\n[![Downloads](https://pepy.tech/badge/kedro-snowflake)](https://pepy.tech/project/kedro-snowflake)\n\n[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=getindata_kedro-snowflake\u0026metric=sqale_rating)](https://sonarcloud.io/summary/new_code?id=getindata_kedro-snowflake)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=getindata_kedro-snowflake\u0026metric=coverage)](https://sonarcloud.io/summary/new_code?id=getindata_kedro-snowflake)\n[![Documentation Status](https://readthedocs.org/projects/kedro-snowflake/badge/?version=latest)](https://kedro-snowflake.readthedocs.io/en/latest/?badge=latest)\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://getindata.com/solutions/ml-platform-machine-learning-reliable-explainable-feature-engineering\"\u003e\u003cimg height=\"150\" src=\"https://getindata.com/img/logo.svg\"\u003e\u003c/a\u003e\n  \u003ch3 align=\"center\"\u003eWe help companies turn their data into assets\u003c/h3\u003e\n\u003c/p\u003e\n\n## About\nThis plugin allows to run full Kedro pipelines in Snowflake. Right now it supports\n* Kedro starter, to get you up to speed fast\n* automatically creating Snowflake Stored Procedures from Kedro nodes (using Snowpark SDK)\n* translating Kedro pipeline into Snowflake tasks graph\n* running Kedro pipeline fully within Snowflake, without external system\n* using Kedro's official `SnowparkTableDataSet`\n* automatically storing intermediate data as Transient Tables (if Snowpark's DataFrames are used)\n* \u003cspan style=\"color:yellow;float:left;margin: 0px 7px 0px 0px\"\u003e**(New!)\u003c/span\u003e\u003c/span\u003e** [MLflow](https://mlflow.org/) integration with Snowflake with examples in _Snowflights_ Kedro starter\n\n## Documentation\nFor detailed documentation refer to https://kedro-snowflake.readthedocs.io/\n\n## Usage\n### With starter\n1. Install the plugin\n    ```bash\n    pip install \"kedro-snowflake\u003e=0.1.0\" \n    ```\n2. Create new project with our Kedro starter ❄️ _Snowflights_ 🚀:\n    ```bash\n    kedro new --starter=snowflights --checkout=master\n    ```\n    \u003cdetails\u003e\n        \u003csummary\u003eAnd answer the interactive prompts ⬇️ (click to expand) \u003c/summary\u003e\n    \n    ```\n    Project Name\n    ============\n    Please enter a human readable name for your new project.\n    Spaces, hyphens, and underscores are allowed.\n     [Snowflights]: \n    \n    Snowflake Account\n    =================\n    Please enter the name of your Snowflake account.\n    This is the part of the URL before .snowflakecomputing.com\n     []: abc-123\n    \n    Snowflake User\n    ==============\n    Please enter the name of your Snowflake user.\n     []: user2137\n    \n    Snowflake Warehouse\n    ===================\n    Please enter the name of your Snowflake warehouse.\n     []: compute-wh\n    \n    Snowflake Database\n    ==================\n    Please enter the name of your Snowflake database.\n     [DEMO]: \n    \n    Snowflake Schema\n    ================\n    Please enter the name of your Snowflake schema.\n     [DEMO]: \n    \n    Snowflake Password Environment Variable\n    =======================================\n    Please enter the name of the environment variable that contains your Snowflake password.\n    Alternatively, you can re-configure the plugin later to use Kedros credentials.yml\n     [SNOWFLAKE_PASSWORD]:       \n    \n    Pipeline Name Used As A Snowflake Task Prefix\n    =============================================\n\n     [default]:\n\n    Enable Mlflow Integration (See Documentation For The Configuration Instructions)\n    ================================================================================\n\n     [False]: \n\n    The project name 'Snowflights' has been applied to: \n    - The project title in /tmp/snowflights/README.md\n    - The folder created for your project in /tmp/snowflights\n    - The project's python package in /tmp/snowflights/src/snowflights\n    ```\n    \u003c/details\u003e\n\n3. Run the project\n    ```bash\n    cd snowflights\n    kedro snowflake run --wait-for-completion\n    ```\n\n### In existing Kedro project\n1. Install the plugin\n    ```bash\n    pip install \"kedro-snowflake\u003e=0.1.0\" \n    ```\n2. Initialize the plugin\n    ```bash\n    kedro snowflake init \u003cACCOUNT\u003e \u003cUSER\u003e \u003cPASSWORD_FROM_ENV\u003e \u003cDATABASE\u003e \u003cSCHEMA\u003e \u003cWAREHOUSE\u003e\n    ```\n3. Run the project\n    ```bash\n    kedro snowflake run --wait-for-completion\n    ```\n   \n### Kedro pipeline in Snowflake Tasks\n\n\u003cimg src=\"./docs/images/kedro-snowflake-tasks-graph.png\" alt=\"Kedro Snowflake Plugin\" title=\"Kedro Snowflake Plugin\" /\u003e\n\nExecution:\n\n\u003cimg src=\"./docs/images/snowflake_running_pipeline.gif\" alt=\"Kedro Snowflake Plugin CLI\" title=\"Kedro Snowflake Plugin CLI\" /\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fkedro-snowflake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgetindata%2Fkedro-snowflake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fkedro-snowflake/lists"}