{"id":31825310,"url":"https://github.com/ngr-t/luigi_for_data_science","last_synced_at":"2025-10-11T15:58:46.774Z","repository":{"id":196817955,"uuid":"88053311","full_name":"ngr-t/luigi_for_data_science","owner":"ngr-t","description":"Extension of luigi.Task more suitable for reproducable data analysis workflows","archived":false,"fork":false,"pushed_at":"2017-09-06T11:35:11.000Z","size":17,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-09-27T19:14:25.696Z","etag":null,"topics":["data-science","luigi-workflows","python","workflow-engine"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ngr-t.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-04-12T13:11:43.000Z","updated_at":"2023-09-27T19:14:35.892Z","dependencies_parsed_at":null,"dependency_job_id":"5ee9f0a5-23da-48f9-b345-ffaf3764a1cc","html_url":"https://github.com/ngr-t/luigi_for_data_science","commit_stats":null,"previous_names":["ngr-t/luigi_for_data_science"],"tags_count":null,"template":null,"template_full_name":null,"purl":"pkg:github/ngr-t/luigi_for_data_science","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ngr-t%2Fluigi_for_data_science","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ngr-t%2Fluigi_for_data_science/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ngr-t%2Fluigi_for_data_science/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ngr-t%2Fluigi_for_data_science/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ngr-t","download_url":"https://codeload.github.com/ngr-t/luigi_for_data_science/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ngr-t%2Fluigi_for_data_science/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279007614,"owners_count":26084336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","luigi-workflows","python","workflow-engine"],"created_at":"2025-10-11T15:58:37.027Z","updated_at":"2025-10-11T15:58:46.770Z","avatar_url":"https://github.com/ngr-t.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Luigi for reproducible data analysis workflow\n========\n\n`luigi` is a very powerful DAG workflow manager with strong extensibility.\nIt's useful to build data analysis pipelines, but some part of its default operation is unfavorable from the view of reproducibility and consistency.\nThe point is that `luigi.Task` by default checks only the existence of output object, therefore it's considered as completed if inputs changed but output object exists. \n\nHere I present an extension of `luigi.Task` more suitable for reproducable data analysis workflows. It override `complete` method of `luigi.Task` as to compare the hash values of inputs to those of previous run.\n\nThanks to the `luigi` team.\n\n\nHow to use?\n-----------\n\n1. Make your tasks inherit `hash_checking_tasks.TaskWithCheckingInputHash`\n2. Make the task's output and all the input inherit `hash_checking_tasks.HashableTarget`.\n3. Run.\n\n\nHow does it work?\n-----------------\n\n`TaskWithCheckingInputHash` is an extension of `luigi.Task` with below operation:\n\n  - check the dependent tasks' completeness in `complete()` method.\n  - check if the input of previous run is equal to that of the current run.\n  - if the run is successful, store the information about the task.\n\n`TaskWithCheckingInputHash` rely on `HashableTarget` that:\n\n  - we can check the equality of the content of targets by comparing the values of `hash_content()`.\n  - we can retrieve the information about the `Task` which made the current output (if exists) by `get_current_input_hash()`\n  - we can store the information about the `Task` which made the output by `store_input_hash()`\n\n\nTODO:\n  - [ ] Docstrings for the whole public methods.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fngr-t%2Fluigi_for_data_science","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fngr-t%2Fluigi_for_data_science","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fngr-t%2Fluigi_for_data_science/lists"}