{"id":13907959,"url":"https://github.com/Comcast/python-batch-runner","last_synced_at":"2025-07-18T06:32:00.647Z","repository":{"id":60722812,"uuid":"186009464","full_name":"Comcast/python-batch-runner","owner":"Comcast","description":"A tiny framework for building batch applications as a collection of tasks in a workflow.","archived":false,"fork":false,"pushed_at":"2021-12-28T02:43:23.000Z","size":636,"stargazers_count":23,"open_issues_count":4,"forks_count":10,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-25T08:24:37.905Z","etag":null,"topics":["batch","batch-processing","microframework","python","tiny-framework","workflow-engine"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Comcast.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-05-10T15:12:23.000Z","updated_at":"2024-06-23T16:21:34.000Z","dependencies_parsed_at":"2022-10-03T21:17:53.274Z","dependency_job_id":null,"html_url":"https://github.com/Comcast/python-batch-runner","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comcast%2Fpython-batch-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comcast%2Fpython-batch-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comcast%2Fpython-batch-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Comcast%2Fpython-batch-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Comcast","download_url":"https://codeload.github.com/Comcast/python-batch-runner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226361621,"owners_count":17612930,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","batch-processing","microframework","python","tiny-framework","workflow-engine"],"created_at":"2024-08-06T23:02:21.134Z","updated_at":"2024-11-25T16:31:15.253Z","avatar_url":"https://github.com/Comcast.png","language":"Python","readme":"# python-batch-runner\n[![Documentation Status](https://readthedocs.org/projects/python-batch-runner/badge/?version=latest)](https://python-batch-runner.readthedocs.io/en/latest/?badge=latest)\n\nFor more complete documentation, please see: https://python-batch-runner.readthedocs.io/\n\npython-batch-runner is a microframework to assist with building small to medium scale batch applications without needing to build the scaffolding from scratch.\n\nIt provides the following with zero or minimal setup required:\n* Basic project directory structure\n* Logging and automatic purging of old log files\n* Email notification upon job completion, with attached log files in case of failure\n* Configurability of task dependencies and parallel execution\n* Self-managed restartability from point of failure and state management\n* Data sharing across tasks\n\n## Installation\n```bash\npip install python-batch-runner\n```\n\n## Usage\n\n### New Project Setup\nA simple setup function is included and can be run by executing:\n```bash\npyrunner --setup\n```\nThis will prompt you for three inputs:\n1. Project Name\n    * Provide a name for your project/application without any spaces (i.e. MySampleProject). If spaces are included, they will be removed.\n2. Project Path\n    * Provide the path to create the project directory in. The project name (lowercased) will be appended to this path.\n    * If left blank, the path will default to the current working directory.\n3. Execution Mode\n    * PyRunner will operate in either SHELL or PYTHON mode.\n    * NOTE: SHELL-only mode will be deprecated in future versions.\n\nUpon completion, a new directory at the provided (or default/current) path will be created, along with minimum necessary subdirectory and files.\n\n### Core Files\nUpon above setup, the following three files will be generated:\n* .../config/app_profile\n  - Contains a list of exports to setup the basic environment prior to execution. This file is sourced before anything else is executed by PyRunner.\n* .../config/\u003cproject_name\u003e.lst\n  - The process list file.\n  - Contains header lines: Line 1 for execution mode (SHELL or PYTHON) and line 2 for column names (line 2 is only for user reference and may be deleted).\n  - Any subsequent lines must be entered by the user. Each line must be a single task (pipe-separated) that describes the following:\n    a. Task ID (must be unique and 1 or higher)\n    b. Parent ID's (comma separated list of numbers that describe which tasks must successfully execute before this task will trigger)\n    c. Maximum # of Retries (0 will mean the task will never run; 3 will mean the task will retry upon failure until executed a maximum of 3 times)\n    d. Task Name\n    e. Task Command (only in SHELL Mode - do not include for PYTHON mode)\n    f. Task Module (only in PYTHON Mode - do not include for SHELL mode)\n    g. Task Worker (only in PYTHON Mode - do not include for SHELL mode)\n    h. Task Arguments (only in PYTHON Mode - do not include for SHELL mode)\n    e. Absolute Path to Log File\n* .../main.sh\n  - For convenience. Simply executes pyrunner with the minimally required options. Any arguments to this script will be passed through to the python script.\n  - Not Required\n\n### Execution Modes\n#### SHELL Mode\n* Will allow any shell command to be executed as a task/process.\n* This provides a free-form mode which can allow you to use scripts and executables from any language.\n* Executes each task as a new subprocess which inherits from the parent environment, but independent of other task environments.\n\n#### PYTHON Mode\n* Restricts execution to only a single class from a user-defined module per task/process.\n* Executes each task as a new thread.\n* This has the benefit of allowing tasks to communicate via a common set of key/value pairs using a Context object. This effectively allows for the storing of state information during execution, and in case of a failure, this state will be preserved for job restarts.\n\n### Execution Options\n| Option | Argument | Description |\n| --- | --- | --- |\n| --env | [variable_name]=[variable_value] | Set environment variable - equivalent to export [variable_name]=[variable_value] |\n| --cvar | [variable_name]=[variable_value] | Set context variable to be available at the start of job. |\n| -r | | Restart flag. Causes PyRunner to check the APP_TEMP_DIR for existing *.ctllog files to restart a job from failure. Fresh run if no *.ctllog file found. |\n| -n *or* --max-procs | integer | Sets the absolute maximum number of parallel processes allowed to run concurrently. |\n| -x *or* --exec-only | comma separated list of process ID's | Executes only the given process ID(s) from the .lst file. |\n| --exec-proc-name | single process name | Similar to --exec-only - Executes only the process ID identified by the given process name. |\n| -A *or* --to *or* --ancestors | single process ID | Executes given process ID and all preceding/ancestor processes. |\n| -D *or* --from *or* --descendents | single process ID | Executes given process ID and all subsequent/descendent processes. |\n| -N *or* --norun | comma separated list of process ID's | Prevents the given process ID(s) from executing. |\n| -e *or* --email | email address | Sets email address to send job notification email after run completion. Overrides **all** other APP_EMAIL settings. |\n| --es *or* --email-on-success | true/false or 1/0 | Enables or disables email notifications when job exits with success. Default is True. |\n| --ef *or* --email-on-fail | true/false or 1/0 | Enables or disables email notifications when job exits with failure. Default is True. |\n| -i *or* --interactive | | Primarily for use with -x option. Launches in interactive mode which will request input from user if a Context variable is not found. |\n| -d *or* --debug | | Debug option that only serves to provide a more detailed output during execution to show names of pending, running, failed, etc. tasks. |\n| --dump-logs | | Enables job to dump to STDOUT logs for all failed tasks after job exits. |\n| --nozip | | Disables zipping of log files after job exits. |\n| -t *or* --tickrate | | Sets the number of checks per second that the execution engine performs to poll running processes. |\n| -h *or* --help | | Prints out options and other details. |\n| -v *or* --version | | Prints out the installed PyRunner version. |\n\n## Contribute\nPlease read the CONTRIBUTING file for more details.\n\n## License\npython-batch-runner is released under the Apache 2.0 License. Please read the LICENSE file for more details.\n","funding_links":[],"categories":["HarmonyOS"],"sub_categories":["Windows Manager"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FComcast%2Fpython-batch-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FComcast%2Fpython-batch-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FComcast%2Fpython-batch-runner/lists"}