{"id":34099979,"url":"https://github.com/tamsanh/kedro-wings","last_synced_at":"2025-12-14T16:07:30.364Z","repository":{"id":47569459,"uuid":"269113608","full_name":"tamsanh/kedro-wings","owner":"tamsanh","description":"Kedro Wings automatically creates catalog entries to simplify Kedro pipeline writing. See the video here: https://www.youtube.com/watch?v=p4ELo1tqbYY","archived":false,"fork":false,"pushed_at":"2022-12-26T20:49:13.000Z","size":97,"stargazers_count":21,"open_issues_count":6,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-09T15:48:12.216Z","etag":null,"topics":["kedro","kedro-catalog","kedro-hook","kedro-plugin","programmatic-kedro"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tamsanh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-03T14:42:29.000Z","updated_at":"2025-10-10T18:19:16.000Z","dependencies_parsed_at":"2023-01-31T01:31:14.960Z","dependency_job_id":null,"html_url":"https://github.com/tamsanh/kedro-wings","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tamsanh/kedro-wings","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamsanh%2Fkedro-wings","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamsanh%2Fkedro-wings/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamsanh%2Fkedro-wings/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamsanh%2Fkedro-wings/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tamsanh","download_url":"https://codeload.github.com/tamsanh/kedro-wings/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tamsanh%2Fkedro-wings/sbom","scorecard":{"id":867349,"data":{"date":"2025-08-11","repo":{"name":"github.com/tamsanh/kedro-wings","commit":"42db6ba22582af49123a3e232df2dd121a0a7dfc"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.7,"checks":[{"name":"Code-Review","score":0,"reason":"Found 2/29 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 3 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"17 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2024-48 / GHSA-fj7x-q9j7-g6q6","Warn: Project is vulnerable to: GHSA-29gw-9793-fvw7","Warn: Project is vulnerable to: PYSEC-2015-24 / GHSA-4vwq-x64q-j4cj","Warn: Project is vulnerable to: PYSEC-2017-46 / GHSA-66gw-5xpf-gfp5","Warn: Project is vulnerable to: PYSEC-2015-25 / GHSA-92mr-v722-f48m","Warn: Project is vulnerable to: PYSEC-2022-12 / GHSA-pq7m-3gw7-gq5x","Warn: Project is vulnerable to: PYSEC-2017-47","Warn: Project is vulnerable to: GHSA-44cc-43rp-5947","Warn: Project is vulnerable to: PYSEC-2021-130 / GHSA-4952-p58q-6crx","Warn: Project is vulnerable to: GHSA-9q39-rmj3-p4r2","Warn: Project is vulnerable to: GHSA-747f-ww56-4q4h","Warn: Project is vulnerable to: GHSA-rm69-wvpv-r2w7","Warn: Project is vulnerable to: PYSEC-2022-43017 / GHSA-qwmp-2cf2-g9g6","Warn: Project is vulnerable to: PYSEC-2013-22 / GHSA-27x4-j476-jp5f","Warn: Project is vulnerable to: PYSEC-2025-49 / GHSA-5rjg-fvgr-3xxf","Warn: Project is vulnerable to: GHSA-cx63-2mw6-8hw5","Warn: Project is vulnerable to: PYSEC-2022-43012 / GHSA-r9hx-vwmv-q579"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-24T03:01:34.622Z","repository_id":47569459,"created_at":"2025-08-24T03:01:34.622Z","updated_at":"2025-08-24T03:01:34.622Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27730980,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-14T02:00:11.348Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kedro","kedro-catalog","kedro-hook","kedro-plugin","programmatic-kedro"],"created_at":"2025-12-14T16:07:29.878Z","updated_at":"2025-12-14T16:07:30.357Z","avatar_url":"https://github.com/tamsanh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kedro Wings\n\nAs Seen on [YouTube DataEngineerOne](https://www.youtube.com/c/DataEngineerOne):  \n* **[Kedro Wings: It's almost too easy to write pipelines this way.](https://www.youtube.com/watch?v=p4ELo1tqbYY)**\n* **[Easy Stateful Pipelines with Chronocoding and Kedro Wings](https://youtu.be/n--1fO-NxC0)**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"255\" src=\"https://github.com/tamsanh/kedro-wings/blob/master/images/kedro-wings.png\"\u003e\n\u003c/p\u003e\n\nGive your next kedro project Wings! The perfect plugin for brand new pipelines, and new kedro users.\nThis plugin enables easy and fast creation of datasets so that you can get straight into coding your pipelines.\n\n## Quick Start Usage Example: Iris Example\n\nThe following example is a recreation of the iris example pipeline.\n\nKedro Wings enables super fast creation of pipelines by taking care of all the catalog work for you.\nCatalog entries are automatically created by parsing the values for your nodes' inputs and outputs.\n\nThis pipeline automatically creates a dataset that reads from the `iris.csv` and then it creates 12 more datasets, corresponding to the outputs and inputs of the other datasets.\n\n```python\nwing_example = Pipeline([\n    node(\n        split_data,\n        inputs=['01_raw/iris.csv', 'params:example_test_data_ratio'],\n        outputs=dict(\n            train_x=\"02_intermediate/example_train_x.csv\",\n            train_y=\"02_intermediate/example_train_y.csv\",\n            test_x=\"02_intermediate/example_test_x.csv\",\n            test_y=\"02_intermediate/example_test_y.csv\")\n        ),\n    node(\n        train_model,\n        [\"02_intermediate/example_train_x.csv\", \"02_intermediate/example_train_y.csv\", \"parameters\"],\n        outputs=\"06_models/example_model.pkl\",\n    ),\n    node(\n        predict,\n        inputs=dict(\n            model=\"06_models/example_model.pkl\",\n            test_x=\"02_intermediate/example_test_x.csv\"\n        ),\n        outputs=\"07_model_output/example_predictions.pkl\",\n    ),\n    node(\n        report_accuracy,\n        inputs=[\"07_model_output/example_predictions.pkl\", \"02_intermediate/example_test_y.csv\"],\n        None\n    ),\n])\n```\n\n## Quick Start Example: Chronocoding\n\nWatch the video on Chronocoding here: **[Easy Stateful Pipelines with Chronocoding and Kedro Wings](https://youtu.be/n--1fO-NxC0)**\n\nSometimes, there arises a need to rewrite data to the same path. This makes it easier to save state between kedro runs.\nUsing KedroWings, you can automatically generate *chronocoded* datasets which temporally separates a read and write to a dataset.\n\nBy adding an `!` at the end of a dataset, we signal to kedro that we wish to overwrite the data in that same filepath. Thus, we get around kedro's DAG requirement for datasets.\n\nIn Depth Breakdown on Chronocoding here: [\\[KED-1667\\] Chronocoding: Solving the Problem of State Tracking with Temporally Sensitive DAGs](https://github.com/quantumblacklabs/kedro/issues/341)  \n\n```python\ndef state_modifier(state: str) -\u003e str:\n    current_value = int(state)\n    new_value = current_value + 1\n    return str(new_value)\n\ndef create_pipelines(**kwargs):\n    return Pipeline([\n        node(\n            state_modifier,\n            inputs=\"01_raw/state.txt\",\n            outputs=\"01_raw/state.txt!\"\n        ),\n    ])\n```\n\n\n## Installation\n\nKedro Wings is available on pypi, and is installed with [kedro hooks](https://kedro.readthedocs.io/en/latest/04_user_guide/15_hooks.html).\n\n\n\n``` console\npip install kedro-wings\n```\n\n\n### Setup with Kedro Pipeline\n\nSimply add a `KedroWings` instance to the `ProjectContext` `hooks` tuple.\n\n```python\nfrom kedro_wings import KedroWings\n\n\nclass ProjectContext(KedroContext):\n    hooks = (\n        KedroWings(),\n    )\n```\n\n### Setup with Jupyter Notebook\n\nSimply pass the kedro context into `KedroWings`, and it will automatically add all catalog entries from all available pipelines.\n\n```python\n# Load the context if not using a kedro jupyter notebook\nfrom kedro.framework.context import load_context\ncontext = load_context('./')\n\n# Pass the context into KedroWings\nfrom kedro_wings import KedroWings\nKedroWings(context=context)\n\n# context catalog now has all wings datasets available.\ncontext.catalog.list()\n```\n\n## Usage\n\n### Catalog Creation\n\nCatalog entries are created using dataset input and output strings. The API is simple:\n\n```python\ninputs=\"[PATH]/[NAME].[EXT]\"\n```\n\nThe `PATH` portion determines the directory where a file will be saved.\nThe `NAME` portion determines the final output name of the file to be saved.\nThe `EXT`  portion determines the dataset used to save and load that particular data.\n\n\n##### Ex: Creating an iris.csv reader\n```python\nnode(split_data, inputs='01_raw/iris.csv', outputs='split_data_output')\n```\n\nThis will create a `pandas.CSVDataSet` pointing at the `01_raw/iris.csv` file.\n\n\n##### Ex: Overwrite a Kedro Wing dataset using `catalog.yml`\n```python\n# pipeline.py\nnode(split_data, inputs='01_raw/iris.csv', outputs='split_data_output')\n```\n\n```yaml\n# catalog.yml\n01_raw/iris.csv':\n    type: pandas.CSVDataSet\n    filepath: data/01_raw/iris.csv\n```\n\nIf a catalog entry already exists inside of `catalog.yml`, with a name that matches the wing catalog name,\nKedroWings will NOT create that catalog, and will instead defer to the `catalog.yml` entry.\n\n\n#### Default Datasets\n\nThe following are the datasets available by default.\n\n```python\ndefault_dataset_configs={\n\".csv\": {\"type\": \"pandas.CSVDataSet\"},\n\".yml\": {\"type\": \"yaml.YAMLDataSet\"},\n\".yaml\": {\"type\": \"yaml.YAMLDataSet\"},\n\".xls\": {\"type\": \"pandas.ExcelDataSet\"},\n\".txt\": {\"type\": \"text.TextDataSet\"},\n\".png\": {\"type\": \"pillow.ImageDataSet\"},\n\".jpg\": {\"type\": \"pillow.ImageDataSet\"},\n\".jpeg\": {\"type\": \"pillow.ImageDataSet\"},\n\".img\": {\"type\": \"pillow.ImageDataSet\"},\n\".pkl\": {\"type\": \"pickle.PickleDataSet\"},\n\".parquet\": {\"type\": \"pandas.ParquetDataSet\"},\n\".json\": {\"type\": \"json.JSONDataSet\"}, # Only available in kedro 0.16.3\n}\n```\n\n### Configuration\n\nKedro Wings supports configuration on instantiation of the hook.\n\n```\nKedroWings(dataset_configs, paths, root, namespaces, enabled, context)\n```\n\n#### dataset_configs\n```\n:param dataset_configs: A mapping of file name extensions to the type of dataset to be created.\n\n```\n\nThis allows the default dataset configurations to be overridden.\nThis also allows the default extension to dataset mapping to be overridden or extended for other datasets.\n\nLonger extensions are prioritized over shorter extensions,\nmeaning multiple encoding methods can be applied to a single filetype.\n\n##### Ex: Make default csv files use pipes as separators\n\n```python\nKedroWings(dataset_configs={\n    '.csv': {'type': 'pandas.CSVDataSet', 'sep': '|'},\n})\n```\n\n##### Ex: Use dataset types directly\n\n```python\nfrom kedro.extras.dataset import pandas\nKedroWings(dataset_configs={\n    '.csv': pandas.CSVDataSet,\n})\n```\n\n##### Ex: Save CSVs with pipes or commas\n\n```python\nfrom kedro.extras.dataset import pandas\nKedroWings(dataset_configs={\n    '.comma.csv': pandas.CSVDataSet,\n    '.pipe.csv': {'type': 'pandas.CSVDataSet', 'sep': '|'},\n})\n```\n\n\n#### paths\n\nThis allows specified paths to be remapped\n\n```\n:param paths: A mapping of old path names to new path names.\n```\n\n##### Ex: Moving data from 06_models to a new_models folder\n\n```python\nKedroWings(paths={\n    '06_models': 'new_models',\n})\n```\n\n#### root\nThis setting is prepended to any paths parsed. This is useful if the dataset supports `fsspec`.\n\n```\n:param root: The root directory to save files to. Default: data\n```\n\n##### Ex: Saving data to s3 instead of the local directory.\n\n```python\nKedroWings(root='s3a://my-bucket/kedro-data')\n```\n\n##### Ex: Allow individual datasets to choose their root\n\n```python\nKedroWings(root=None)\n```\n\n#### namespaces\n\nNamespaces from modular pipelines are supported.\nThis parameter should be a list of the namespaces that KedroWings should account for.\nIf a namespace is encountered, the output filepath will include the namespace in the extension.\n\n##### Ex: Namespace\n\nThe determined file paths would be `iris.example1.csv` and `iris2.example2.csv`.\n\n```python\nKedroWings(namespaces=['example1'])\n\npipeline(Pipeline([node(lambda x: x, inputs='iris.csv', outputs='iris2.csv')]), namespace=\"example1\")\n```\n\n#### enabled\nThis setting allows easy enabling and disabling of the plugin.\n\n```\n:param enabled: Convenience flag to enable or disable this plugin.\n```\n\n##### Ex: Use an environment variable to enable or disable wings\n\n```python\nKedroWings(enabled=os.getenv('ENABLE_WINGS'))\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftamsanh%2Fkedro-wings","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftamsanh%2Fkedro-wings","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftamsanh%2Fkedro-wings/lists"}