{"id":17383628,"url":"https://github.com/mortoray/shelljob","last_synced_at":"2025-04-15T09:53:45.598Z","repository":{"id":43723951,"uuid":"282459347","full_name":"mortoray/shelljob","owner":"mortoray","description":"Python process and file system module","archived":false,"fork":false,"pushed_at":"2022-12-08T07:45:33.000Z","size":294,"stargazers_count":6,"open_issues_count":4,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-28T19:07:41.410Z","etag":null,"topics":["capture","parallel","python","shell-script","subprocesses"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mortoray.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-25T14:23:57.000Z","updated_at":"2022-02-21T17:00:22.000Z","dependencies_parsed_at":"2022-08-22T13:01:08.740Z","dependency_job_id":null,"html_url":"https://github.com/mortoray/shelljob","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortoray%2Fshelljob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortoray%2Fshelljob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortoray%2Fshelljob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mortoray%2Fshelljob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mortoray","download_url":"https://codeload.github.com/mortoray/shelljob/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249048716,"owners_count":21204306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["capture","parallel","python","shell-script","subprocesses"],"created_at":"2024-10-16T07:43:19.401Z","updated_at":"2025-04-15T09:53:45.579Z","avatar_url":"https://github.com/mortoray.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# shelljob\n\nThis provides a clean way to execute subprocesses, either one or multiple in parallel, capture their output and monitor progress:\n\n- Single sub process `call` with optional timeout\n- High level `FileMonitor` to execute several processes in parallel and store output in a file\n- Low level `Group` execution to execute jobs in parallel and capture output\n\nAdditional tools for working with the filesystem are also included: \n\n- `find` which offers much of the functionality of the shell find utility\n- `shelljob.fs.NamedTempFile` provides a _with_ block wrapper for temporary named files\n\n[API Documentation](https://mortoray.github.io/shelljob/)\n\nI use this actively in my own projects, such as my online escape games at  [Edaqa's Room](https://edaqasroom.com/). I've not needed any new features in a while -- it's a fairly stable package.\n\n# Install\n\n```\npip install shelljob\n```\n\n# Parallel Subprocesses\n\nUsing the Job system is the quickest approach to just run processes and log their output (by default in files named '/tmp/job_ID.log')\n\n````\nfrom shelljob import job\n\njm = job.FileMonitor()\njm.run([\n\t[ 'ls', '-alR', '/usr/local' ],\n\t'my_prog',\n\t'build output input',\n])\n````\n\nAn array will passed directly to `subprocess.Popen`, a string is first parsed with `shlex.split`.\n\nThe lower level `Group` class provides a simple container for more manual job management.\n\n````\nfrom shelljob import proc\n\ng = proc.Group()\np1 = g.run( [ 'ls', '-al', '/usr/local' ] )\np2 = g.run( [ 'program', 'arg1', 'arg2' ] )\n\nwhile g.is_pending():\n\tlines = g.readlines()\n\tfor proc, line in lines:\n\t\tsys.stdout.write( \"{}:{}\".format( proc.pid, line ) )\n````\n\n## Encoding\n\nBy default the output will be binary encoding. You can specify an `encoding='utf-8'` to the `run` command to use an encoded text stream instead. Be aware that if the encoding fails (the program emits an invalid sequence) the running will be interrupted. You should also use the `on_error` function to check for this.\n\nLine-endings will always be preserved.\n\n# Simple Subprocess calls\n\nA simplified `call` function allows timeouts on subprocesses and easy acces to their output.\n\n````\nfrom shelljob import proc\n\n# capture the output\noutput = proc.call( 'ls /tmp' )\n# this raises a proc.Timeout exception\nproc.call( 'sleep 10', timeout = 0.1 )\n````\n\n# Find\n\nThe 'find' funtion is a multi-faceted approach to generating listings of files.\n\n````\nfrom shelljob import fs\n\nfiles = fs.find( '/usr/local', name_regex = '.*\\\\.so' )\nprint( \"\\n\".join(files) )\n````\n\nRefer to the [API docs](https://mortoray.github.io/shelljob/) for all parameters. Just let me know if there is some additional option you need.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmortoray%2Fshelljob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmortoray%2Fshelljob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmortoray%2Fshelljob/lists"}