{"id":19120695,"url":"https://github.com/waylonwalker/steel-toes","last_synced_at":"2025-05-05T16:20:18.263Z","repository":{"id":105815309,"uuid":"266452577","full_name":"WaylonWalker/steel-toes","owner":"WaylonWalker","description":"a kedro hook to protect against breaking changes to data","archived":false,"fork":false,"pushed_at":"2023-12-04T20:09:08.000Z","size":938,"stargazers_count":9,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-05T16:20:06.420Z","etag":null,"topics":["cli","data","kedro","kedro-hook","kedro-plugin","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WaylonWalker.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"contributing.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"WaylonWalker","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2020-05-24T02:01:37.000Z","updated_at":"2023-02-15T16:32:27.000Z","dependencies_parsed_at":"2025-04-19T02:32:05.564Z","dependency_job_id":null,"html_url":"https://github.com/WaylonWalker/steel-toes","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WaylonWalker%2Fsteel-toes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WaylonWalker%2Fsteel-toes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WaylonWalker%2Fsteel-toes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WaylonWalker%2Fsteel-toes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WaylonWalker","download_url":"https://codeload.github.com/WaylonWalker/steel-toes/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252531879,"owners_count":21763293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","data","kedro","kedro-hook","kedro-plugin","python"],"created_at":"2024-11-09T05:14:44.000Z","updated_at":"2025-05-05T16:20:18.237Z","avatar_url":"https://github.com/WaylonWalker.png","language":"Python","funding_links":["https://github.com/sponsors/WaylonWalker"],"categories":[],"sub_categories":[],"readme":"\u003ch1 align='center'\u003e Steel Toes\u003c/h1\u003e\n\n\u003cimg src=\"https://user-images.githubusercontent.com/22648375/218914190-22fb1188-5587-4152-ae46-6fe7cb770ca2.png\" alt=\"steel-toes\" width=\"250\" align=right\u003e\n\n_a kedro hook to protect against breaking changes to data_\n\n[steel-toes](https://steel-toes.kedro.dev/) is a kedro hook designed to prevent stepping on your teammates\ntoes. It will branch your data automatically based on your git branch, or\nmanually by passing the branch name into the hook.\n\n\u003e see [docs](https://steel-toes.kedro.dev/)\n\n## Motivation\n\n`kedro` is a ✨ fantastic project that allows for super-fast prototyping of\ndata pipelines, while yielding production-ready pipelines. `kedro` promotes\ncollaborative projects by giving each team member access to the exact same\ndata. Team members will often make their own branch of the project and begin\nwork. Sometimes these changes will break existing functionality. Sometimes we\nmake mistakes as we develop, and fix them before merging in. Either case can be\ndetrimental to a teammate working downstream of your changes if not careful.\n\n### 🥼 Wear the proper PPE during feature development\n\n`steel-toes` hooks into your catalog to prevent changing downstream data on\nyour teammates while developing in parallel.\n\n### on_catalog_created and before_pipeline_run\n\nWhen your project creates a catalog `steel-toes` will look to see if branched\ndata exists, if it does it will swap the filepath to the branched path. So you\nwill be able to load the latest data from the perspective of any branch\nsimulaneusly.\n\n### after_node_run\n\nAfter your node is ran, before saving, `steel-toes` will check if your\n`filepath` was swapped, if not it will swap it to the branched `filepath`\nbefore saving.\n\n## Installation\n\n`steel-toes` is deployed to pypi and can be `pip` installed.\n\n```console\npip install steel-toes\n```\n\nFor a real kedro project you should add to your requirements.\n\n## Setup\n\nTo add `SteelToes` to your kedro\u003e0.18.0 project add an instance of the\n`SteelToes` hook to your tuple of hooks in src/\u003cproject_name\u003e/settings.py.\n\n```python\n# settings.py\nfrom steel_toes import SteelToes\n\nHOOKS = (SteelToes(),)\n```\n\n### ignore_types\n\nSome datasets have a `_filepath` attribute that is not meant for saving\ndatasets to and is not needed to be \"branched\", and should be ignored from\nsteel_toes, for example `SQLQueryDataSet`.\n\n```python\n# settings.py\nfrom kedro.extras.datasets.pandas.sql_dataset import SQLQueryDataSet, SQLTableDataSet\nfrom steel_toes import SteelToes\n\nHOOKS = (SteelToes(ignore_types=[SQLQueryDataSet, SQLTableDataSet]),)\n```\n\n## Automatic branch naming\n\n`steel_toes` will automatically get the branch name from your git branch.\n\n## Override with environment variable\n\nIn certain situations such as using `kedro docker` in production, there is no\ngit branch to pull from. Setting an environment variable before `steel-toes`\ninitializes will set the branch.\n\n### set environment variable in the shell\n\n```bash\nSTEEL_TOES_BRANCH='PROD'\n```\n\n### set environment variable with python\n\n```bash\nimport os\n\nos.environ[\"STEEL_TOES_BRANCH\"] = \"PROD\"\n```\n\n## Example filenames\n\nHere is an example of what filepaths look like when I add parquet catalog\nentries to the spaceflights project, `steel_toes` will add the branch name\nautomatically just before the file extension.\n\n```bash\nSTEEL-TOES |6 DATASETS PROETECTED\nX_test: /home/waylon/git/spaceflights/data/X_test_main.pq\nX_train: /home/waylon/git/spaceflights/data/X_train_main.pq\npreprocessed_companies: /home/waylon/git/spaceflights/data/02_intermediate/preprocessed_companies_main.pq\npreprocessed_shuttles: /home/waylon/git/spaceflights/data/02_intermediate/preprocessed_shuttles_main.pq\nmodel_input_table: /home/waylon/git/spaceflights/data/03_primary/model_input_table_main.pq\nregressor: /home/waylon/git/spaceflights/data/06_models/regressor_main.pickle\n```\n\n## Logs on first run\n\nWhen first running your pipeline with `steel-toes` it will start the\n`_filepath` swap **after_node_run**, since the swapped file does not yet exist.\n\n\u003e At this point catalog.load('preprocessed_shuttles') will **not** load the\n\u003e branched dataset.\n\n```bash\n❯ kedro run\nINFO     Kedro project spaceflights                                                               session.py:340\n...\nINFO     STEEL_TOES:after_node_run 'preprocessed_shuttles.pq' -\u003e 'preprocessed_shuttles_main.pq'  steel_toes.py:102\n...\nINFO     Completed 6 out of 6 tasks                                                               sequential_runner.py:85\nINFO     Pipeline execution completed successfully.                                               runner.py:90\n```\n\n## Logs after dataset exists\n\nSubsequent runs of kedro will swap the dataset to the branched filepath\nimmediately after the catalog has been created.\n\n\u003e Now catalog.load('preprocessed_shuttles') **will** load the branched dataset.\n\n````bash\nINFO     Kedro project spaceflights                                                                      session.py:340\n...\nINFO     STEEL_TOES:after_catalog_created 'preprocessed_shuttles.pq' -\u003e 'preprocessed_shuttles_main.pq'  steel_toes.py:102\n...\nINFO     Completed 6 out of 6 tasks                                                                      sequential_runner.py:85\nINFO     Pipeline execution completed successfully.                                                      runner.py:90\n\n### CLI Usage\n\nThe CLI provides a handy interface to clean up your branched datasets.\n\n```bash\n$ steel-toes --help\nUsage: steel-toes [OPTIONS] COMMAND [ARGS]...\n\n  help\n\nOptions:\n  -V, --version  Prints version and exits\n  --help         Show this message and exit.\n\nCommands:\n  clean-branch  finds branch datasets and removes them\n````\n\n`steel-toes` also registers itself as a `kedro` global cli plugin. You can run `kedro clean-branch` to clean your branched data.\n\n```bash\n$ kedro clean-branch --help\nUsage: kedro clean-branch [OPTIONS]\n\n  finds branch datasets and removes them\n\nOptions:\n  --dryrun                   Displays the files that would be deleted using\n                             the specified command without actually deleting\n                             them.\n\n  -b, --branch TEXT          git branch to clean files from\n  -h, --help                 Show this message and exit.\n```\n\n## Cleaning up old branches\n\nTo clean up your current branch, running `kedro clean-branch` will remove all\nthe datasets that have been swapped to the current branch. Adding `--dryrun`\nwill only log what `steel-toes` intends to do, and will not delete.\n\n```\n❯ kedro clean-branch --dryrun\nINFO     STEEL_TOES:after_catalog_created 'preprocessed_shuttles.pq' -\u003e 'preprocessed_shuttles_main.pq'                                         steel_toes.py:102\n...\nINFO     STEEL_TOES:dryrun-remove | '/home/waylon/git/spaceflights/data/02_intermediate/preprocessed_shuttles_main.pq'                          steel_toes.py:141\n```\n\nDropping the `--dryrun` flag will delete all the branched datasets.\n\n```\n❯ kedro clean-branch\nINFO     STEEL_TOES:after_catalog_created 'preprocessed_shuttles.pq' -\u003e 'preprocessed_shuttles_main.pq'                                         steel_toes.py:102\n...\nINFO     STEEL_TOES:deleting | '/home/waylon/git/spaceflights/data/02_intermediate/preprocessed_shuttles_main.pq'                          steel_toes.py:141\n```\n\n## Disable\n\nYou can disable `steel-toes` by setting the `STEEL_TOES_ENABLED` environment\nvariable to `False`. This might be useful for debugging inside an environment\nthat you cannot easily make a code change to.\n\n**Mac/Linux**\n\n```\nexport STEEL_TOES_ENABLED=False\n```\n\n**Windows**\n\n```\nset STEEL_TOES_ENABLED=False\n```\n\n## Contributing\n\n**You're Awesome** for considering a contribution! Contributions are welcome,\nplease check out the [Contributing\nGuide](https://github.com/WaylonWalker/steel-toes/blob/main/contributing.md)\nfor more information. Please be a positive member of the community and embrace\nfeedback\n\n## Versioning\n\nWe use [SemVer](https://semver.org/) for versioning. For the versions\navailable, see the [tags on this repository](./tags).\n\n## Author\n\n[![Waylon Walker](https://avatars1.githubusercontent.com/u/22648375?s=120\u0026v=4)](https://github.com/WaylonWalker) - Waylon Walker - _Original Author_\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](./LICENSE).\nfile for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaylonwalker%2Fsteel-toes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwaylonwalker%2Fsteel-toes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwaylonwalker%2Fsteel-toes/lists"}