{"id":18655879,"url":"https://github.com/michaelosthege/gittrail","last_synced_at":"2025-11-05T19:30:32.809Z","repository":{"id":57434873,"uuid":"433400567","full_name":"michaelosthege/gittrail","owner":"michaelosthege","description":"Context manager for enforcing links between data pipeline outputs and git history.","archived":false,"fork":false,"pushed_at":"2022-02-22T16:46:24.000Z","size":42,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-28T05:37:07.691Z","etag":null,"topics":["data-lineage","data-science"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michaelosthege.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-30T11:13:36.000Z","updated_at":"2021-12-29T21:39:06.000Z","dependencies_parsed_at":"2022-09-04T15:32:31.244Z","dependency_job_id":null,"html_url":"https://github.com/michaelosthege/gittrail","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelosthege%2Fgittrail","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelosthege%2Fgittrail/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelosthege%2Fgittrail/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaelosthege%2Fgittrail/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michaelosthege","download_url":"https://codeload.github.com/michaelosthege/gittrail/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239470662,"owners_count":19644163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-lineage","data-science"],"created_at":"2024-11-07T07:20:38.681Z","updated_at":"2025-11-05T19:30:32.748Z","avatar_url":"https://github.com/michaelosthege.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PyPI version](https://img.shields.io/pypi/v/gittrail)](https://pypi.org/project/gittrail)\n[![pipeline](https://github.com/michaelosthege/gittrail/workflows/pipeline/badge.svg)](https://github.com/michaelosthege/gittrail/actions)\n[![coverage](https://codecov.io/gh/michaelosthege/gittrail/branch/main/graph/badge.svg)](https://codecov.io/gh/michaelosthege/gittrail)\n\n\n# `gittrail` - Linking data pipeline outputs to git history\nVersioning of code with git is easy, versioning data pipeline inputs/outputs is hard.\n\n``GitTrail`` helps you to maintain a traceable data lineage by enforcing a\nlink between data files and the commit history of your processing code.\n\nLike blockchain, but easier.\n\n## How it works\n``GitTrail`` is used as a context manager around the code that executes your data processing:\n\n```python\nwith GitTrail(\n    repo=\"/path/to/my_data_processing_code\",\n    data=\"/path/to/my_data_storage\",\n):\n    # TODO: download the pipeline inputs to [data]\n```\n\nInbetween GitTrail sessions you may edit your pipeline code, make commits etc.\n\nWhen your next data processing stage is ready:\n\n```python\nwith GitTrail(\n    repo=\"/path/to/my_data_processing_code\",\n    data=\"/path/to/my_data_storage\",\n):\n    # TODO: run data analysis on inputs from [data]\n    # TODO: save results to [data]\n```\n\nUpon entering the context ``GitTrail`` attaches a log handler to re-route all logging into a `*.log` file in a subdirectory of [data].\nWhen the context exits, the logger is detached and session metadata is stored in a `*.json` file.\nThe metadata includes the current git commit of your [repo], as well MD5 hashes of the files inside [data].\n\nWithin the context, the following two rules are enforced:\n1. The working tree of your code [repo] must be clean (no uncommitted changes).\n2. All files currently found in [data] must have been created/changed in a previous ``GitTrail`` context.\n\nTaken together this means that:\n* You're not allowed to add/edit/anything in [data] by hand.\n* Your data processing code may continue to evolve as you're moving forward through your pipeline.\n* You can amend/rewind/rewrite git commits of your processing code, but the corresponding files in [data] and the audit trail session file must be deleted.\n* All files in the [data] are linked to the processing code that produced them.\n\n## Limitations\n``GitTrail`` can't police everything, so keep the following in mind:\n- Data outside of [data], for example a database, is not tracked.\n    If you're reading/writing data outside of [data] think about how you can trace that in your git history and/or [data] audit trail.\n- Code outside of [repo] is not tracked.\n    Unless your [repo] specifies exact dependency versions, your code may not be 100 % reproducible.\n- Audit trail files are not cryptographically signed, so if you mess with them that's not tracked.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelosthege%2Fgittrail","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaelosthege%2Fgittrail","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelosthege%2Fgittrail/lists"}