{"id":28666114,"url":"https://github.com/python-useful-helpers/exec-helpers","last_synced_at":"2025-10-11T13:06:19.878Z","repository":{"id":57427532,"uuid":"125676721","full_name":"python-useful-helpers/exec-helpers","owner":"python-useful-helpers","description":"Helpers for dealing with python.subprocess.Popen and paramiko.","archived":false,"fork":false,"pushed_at":"2025-05-08T06:58:39.000Z","size":1646,"stargazers_count":18,"open_issues_count":2,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-08T07:32:19.317Z","etag":null,"topics":["paramiko","python","python2","python3","subprocess"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/python-useful-helpers.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-03-17T23:17:15.000Z","updated_at":"2025-05-08T06:58:41.000Z","dependencies_parsed_at":"2024-01-02T09:31:49.415Z","dependency_job_id":"440c5654-90f2-4597-bcfe-8504e47f9690","html_url":"https://github.com/python-useful-helpers/exec-helpers","commit_stats":null,"previous_names":[],"tags_count":168,"template":false,"template_full_name":null,"purl":"pkg:github/python-useful-helpers/exec-helpers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/python-useful-helpers%2Fexec-helpers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/python-useful-helpers%2Fexec-helpers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/python-useful-helpers%2Fexec-helpers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/python-useful-helpers%2Fexec-helpers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/python-useful-helpers","download_url":"https://codeload.github.com/python-useful-helpers/exec-helpers/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/python-useful-helpers%2Fexec-helpers/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259659673,"owners_count":22891678,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["paramiko","python","python2","python3","subprocess"],"created_at":"2025-06-13T14:11:28.500Z","updated_at":"2025-10-11T13:06:14.823Z","avatar_url":"https://github.com/python-useful-helpers.png","language":"Python","readme":"exec-helpers\n============\n\n.. image:: https://github.com/python-useful-helpers/exec-helpers/workflows/Python%20package/badge.svg\n    :target: https://github.com/python-useful-helpers/exec-helpers/actions\n.. image:: https://readthedocs.org/projects/exec-helpers/badge/?version=latest\n    :target: https://exec-helpers.readthedocs.io/\n    :alt: Documentation Status\n.. image:: https://img.shields.io/pypi/v/exec-helpers.svg\n    :target: https://pypi.python.org/pypi/exec-helpers\n.. image:: https://img.shields.io/pypi/pyversions/exec-helpers.svg\n    :target: https://pypi.python.org/pypi/exec-helpers\n.. image:: https://img.shields.io/pypi/status/exec-helpers.svg\n    :target: https://pypi.python.org/pypi/exec-helpers\n.. image:: https://img.shields.io/github/license/python-useful-helpers/exec-helpers.svg\n    :target: https://raw.githubusercontent.com/python-useful-helpers/exec-helpers/master/LICENSE\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n    :target: https://github.com/ambv/black\n\nExecution helpers for simplified usage of subprocess and ssh.\nWhy another subprocess wrapper and why no clear `paramiko`?\n\nHistorically `paramiko` offers good ssh client, but with specific limitations:\nyou can call command with timeout, but without receiving return code,\nor call command and wait for return code, but without timeout processing.\n\nIn the most cases, we are need just simple SSH client with comfortable API for calls, calls via SSH proxy and checking return code/stderr.\nThis library offers this functionality with deadlock free polling and friendly result objects\n(with inline decoding of XML Element tree, YAML, JSON, binary or just strings).\nIn addition this library offers the same API for subprocess calls, but with specific limitation: no parallel calls\n(for protection from race conditions).\n\nPros:\n\n* STDOUT and STDERR polling during command execution - no deadlocks.\n* The same API for subprocess and ssh.\n* Free software: Apache license\n* Open Source: https://github.com/python-useful-helpers/exec-helpers\n* PyPI packaged: https://pypi.python.org/pypi/exec-helpers\n* Self-documented code: docstrings with types in comments\n* Tested: see badges on top\n* Support multiple Python versions:\n\n::\n\n    Python 3.8\n    Python 3.9\n    Python 3.10\n    Python 3.11\n    Python 3.12\n\nThis package includes:\n\n* `SSHClient` - historically the first one helper, which used for SSH connections.\n  Several API calls for sFTP also presents.\n\n* `SSHAuth` - class for credentials storage. `SSHClient` does not store credentials as-is, but uses `SSHAuth` for it.\n  Objects of this class can be copied between ssh connection objects, also it used for `execute_through_host`.\n\n* `Subprocess` - `subprocess.Popen` wrapper with timeouts, polling and almost the same API, as `SSHClient`\n  (except specific flags, like `cwd` for subprocess and `get_tty` for ssh).\n\n* `async_api.Subprocess` - the same, as `Subprocess` helper, but works with asyncio.\n  .. note:: for Windows `ProactorEventLoop` or another non-standard event loop should be used!\n\n* `ExecResult` - class for execution results storage.\n  Contains exit code, stdout, stderr and getters for decoding as JSON, YAML, XML (and LXML) element tree, string, bytearray\n  and brief strings (up to 7 lines).\n\n* `ExitCodes` - enumerator for standard Linux exit codes. BASH return codes (produced from signal codes) also available.\n\nInstallation\n============\n\nStandard: `pip install exec-helpers`\nExtras:\n\n* ``yaml`` - install `PyYaml` for yaml decoding (`PyYAML` is main decoder, `ruamel.YAML` also supported as fallback.)\n\n* ``xml`` - install `defusedxml` for safe XML parsing to `xml.etree.ElementTree.Element`.\n\n* ``lxml`` - install `lxml` for advanced XML parsing. Can be unsafe.\n\n* ``ALL_FORMATS`` (``all-formats``) - install all parsers. When new parsers will be added, it will ne also supported.\n\nUsage\n=====\n\nSSHClient\n---------\n\nBasic initialization of `SSHClient` can be done without construction of specific objects:\n\n.. code-block:: python\n\n    client = exec_helpers.SSHClient(host, username=\"username\", password=\"password\")\n\nIf ssh agent is running - keys will be collected by paramiko automatically,\nbut if keys are in specific location  - it should be loaded manually and provided as iterable object of `paramiko.PKey`.\n\nFor advanced cases or re-use of credentials, `SSHAuth` object should be used.\nIt can be collected from connection object via property `auth`.\n\nCreation from scratch:\n\n.. code-block:: python\n\n    auth = exec_helpers.SSHAuth(\n        username='username',  # str | None\n        password='password',  # str | None\n        key=None,  # type: paramiko.PKey | None\n        keys=None,  # type: Iterable[paramiko.PKey] | None\n        key_filename=None,  # type: list[str] | None\n        passphrase=None,  # str | None\n    )\n\nKey is a main connection key (always tried first) and keys are alternate keys.\nKey filename is a filename or list of filenames with keys, which should be loaded.\nPassphrase is an alternate password for keys, if it differs from main password.\nIf main key now correct for username - alternate keys tried, if correct key found - it became main.\nIf no working key - password is used and None is set as main key.\n\nContext manager is available, connection is closed and lock is released on exit from context.\n\n.. note:: context manager is strictly not recommended in scenarios with fast reconnect to the same host with te same credentials.\n\nSubprocess\n----------\n\nContext manager is available, subprocess is killed and lock is released on exit from context.\n\nBase methods\n------------\nMain methods are `execute`, `check_call` and `check_stderr` for simple executing, executing and checking return code\nand executing, checking return code and checking for empty stderr output.\nThis methods are almost the same for `SSHClient` and `Subprocess`, except specific flags.\n\n.. note:: By default ALL methods have timeout 1 hour, infinite waiting can be enabled, but it's special case.\n\n.. code-block:: python\n\n    result: ExecResult = helper.execute(\n        command,  # type: str | Iterable[str]\n        verbose=False,  # type: bool\n        timeout=1 * 60 * 60,  # type: int | float | None\n        # Keyword only:\n        log_mask_re=None,  # str | None\n        stdin=None,  # type: bytes | str | bytearray | None\n        open_stdout=True,  # type: bool\n        log_stdout=True,  # type: bool\n        open_stderr=True,  # type: bool\n        log_stderr=True,  # type: bool\n        **kwargs\n    )\n\n\n.. code-block:: python\n\n    result: ExecResult = helper.check_call(\n        command,  # type: str | Iterable[str]\n        verbose=False,  # type: bool\n        timeout=1 * 60 * 60,  # type: type: int | float | None\n        error_info=None,  # str | None\n        expected=(0,),  # type: Iterable[int | ExitCodes]\n        raise_on_err=True,  # type: bool\n        # Keyword only:\n        log_mask_re=None,  # str | None\n        stdin=None,  # type: bytes | str | bytearray | None\n        open_stdout=True,  # type: bool\n        log_stdout=True,  # type: bool\n        open_stderr=True,  # type: bool\n        log_stderr=True,  # type: bool\n        exception_class=CalledProcessError,  # type[CalledProcessError]\n        **kwargs\n    )\n\n.. code-block:: python\n\n    result: ExecResult = helper.check_stderr(\n        command,  # type: str | Iterable[str]\n        verbose=False,  # type: bool\n        timeout=1 * 60 * 60,  # type: type: int | float | None\n        error_info=None,  # str | None\n        raise_on_err=True,  # type: bool\n        # Keyword only:\n        expected=(0,),  # Iterable[int | ExitCodes]\n        log_mask_re=None,  # str | None\n        stdin=None,  # type: bytes | str | bytearray | None\n        open_stdout=True,  # type: bool\n        log_stdout=True,  # type: bool\n        open_stderr=True,  # type: bool\n        log_stderr=True,  # type: bool\n        exception_class=CalledProcessError,  # type[CalledProcessError]\n    )\n\n.. code-block:: python\n\n    result: ExecResult = helper(  # Lazy way: instances are callable and uses `execute`.\n        command,  # type: str | Iterable[str]\n        verbose=False,  # type: bool\n        timeout=1 * 60 * 60,  # type: int | float | None\n        # Keyword only:\n        log_mask_re=None,  # str | None\n        stdin=None,  # type: bytes | str | bytearray | None\n        open_stdout=True,  # type: bool\n        log_stdout=True,  # type: bool\n        open_stderr=True,  # type: bool\n        log_stderr=True,  # type: bool\n        **kwargs\n    )\n\n.. note::\n\n  If command is provided as `Iterable[str]`, `shell=True` will be still used,\n  but all command components will be joined with escaping to protect from shell processing.\n\nIf no STDOUT or STDERR required, it is possible to disable this FIFO pipes via `**kwargs` with flags `open_stdout=False` and `open_stderr=False`.\n\nThe next command level uses lower level and kwargs are forwarded, so expected exit codes are forwarded from `check_stderr`.\nImplementation specific flags are always set via kwargs.\n\nIf required to mask part of command from logging, `log_mask_re` attribute can be set global over instance or provided with command.\nAll regex matched groups will be replaced by `'\u003c*masked*\u003e'`.\n\n.. code-block:: python\n\n    result: ExecResult = helper.execute(\n        command=\"AUTH='top_secret_key'; run command\",  # type: str | Iterable[str]\n        verbose=False,  # type: bool\n        timeout=1 * 60 * 60,  # type: Optional[int]\n        log_mask_re=r\"AUTH\\s*=\\s*'(\\w+)'\"  # str | None\n    )\n\n`result.cmd` will be equal to `AUTH='\u003c*masked*\u003e'; run command`\n\nExecResult\n----------\n\nExecution result object has a set of useful properties:\n\n* `cmd` - Command\n* `exit_code` - Command return code. If possible to decode using enumerators for Linux -\u003e it used.\n* `ok` -\u003e `bool`. Command return code is 0 (EX_OK).\n* `stdin` -\u003e `str`. Text representation of stdin.\n* `stdout` -\u003e `tuple[bytes]`. Raw stdout output.\n* `stderr` -\u003e `tuple[bytes]`. Raw stderr output.\n* `stdout_bin` -\u003e `bytearray`. Binary stdout output.\n* `stderr_bin` -\u003e `bytearray`. Binary stderr output.\n* `stdout_str` -\u003e `str`. Text representation of output.\n* `stderr_str` -\u003e `str`. Text representation of output.\n* `stdout_brief` -\u003e `str`. Up to 7 lines from stdout (3 first and 3 last if \u003e7 lines).\n* `stderr_brief` -\u003e `str`. Up to 7 lines from stderr (3 first and 3 last if \u003e7 lines).\n\n* `stdout_json` - STDOUT decoded as JSON.\n\n* `stdout_yaml` - STDOUT decoded as YAML. Accessible only if `PyYAML` or `ruamel.YAML` library installed.\n  (Extras: ``yaml``)\n\n* `stdout_xml` - STDOUT decoded as XML to `ElementTree` using `defusedxml` library. Accessible only if `defusedxml` library installed.\n  (Extras: ``xml``)\n\n* `stdout_lxml` - STDOUT decoded as XML to `ElementTree` using `lxml` library. Accessible only if `lxml` library installed.\n  (Extras: ``lxml``) Can be insecure.\n\n* `timestamp` -\u003e `Optional(datetime.datetime)`. Timestamp for received exit code.\n\nSSHClient specific\n------------------\n\nSSHClient commands support get_pty flag, which enables PTY open on remote side.\nPTY width and height can be set via keyword arguments, dimensions in pixels are always 0x0.\n\nPossible to call commands in parallel on multiple hosts if it's not produce huge output:\n\n.. code-block:: python\n\n    results: dict[tuple[str, int], ExecResult] = SSHClient.execute_together(\n        remotes,  # type: Iterable[SSHClient]\n        command,  # type: str | Iterable[str]\n        timeout=1 * 60 * 60,  # type: type: int | float | None\n        expected=(0,),  # type: Iterable[int | ExitCodes]\n        raise_on_err=True,  # type: bool\n        # Keyword only:\n        stdin=None,  # type: bytes | str | bytearray | None\n        open_stdout=True,  # type: bool\n        open_stderr=True,  # type: bool\n        log_mask_re=None,  # str | None\n        exception_class=ParallelCallProcessError  # type[ParallelCallProcessError]\n    )\n    results  # type: dict[tuple[str, int], exec_result.ExecResult]\n\nResults is a dict with keys = (hostname, port) and and results in values.\nBy default execute_together raises exception if unexpected return code on any remote.\n\nTo open new connection using current as proxy is accessible method `proxy_to`. Basic usage example:\n\n.. code-block:: python\n\n    conn: SSHClient = client.proxy_to(host, username=\"username\", password=\"password\")\n\n.. note:: for full command API please rely API documentation.\n\nFor execute through SSH host can be used `execute_through_host` method:\n\n.. code-block:: python\n\n    result: ExecResult = client.execute_through_host(\n        hostname,  # type: str\n        command,  # type: str | Iterable[str]\n        # Keyword only:\n        auth=None,  # type: SSHAuth | None\n        port=22,  # type: int\n        timeout=1 * 60 * 60,  # type: type: int | float | None\n        verbose=False,  # type: bool\n        stdin=None,  # type: bytes | str | bytearray | None\n        open_stdout=True,  # type: bool\n        log_stdout=True,  # type: bool\n        open_stderr=True,  # type: bool\n        log_stderr=True,  # type: bool\n        log_mask_re=None,  # str | None\n        get_pty=False,  # type: bool\n        width=80,  # type: int\n        height=24  # type: int\n    )\n\nWhere hostname is a target hostname, auth is an alternate credentials for target host.\n\nSSH client implements fast sudo support via context manager:\n\n.. note:: In case of combination sudo + chroot, chroot will be applied first. For alternative order write command with chroot manually.\n\nCommands will be run with sudo enforced independently from client settings for normal usage:\n\n.. code-block:: python\n\n    with client.sudo(enforce=True):\n        ...\n\n\nCommands will be run *without sudo* independently from client settings for normal usage:\n\n.. code-block:: python\n\n    with client.sudo(enforce=False):\n        ...\n\n\"Permanent client setting\":\n\n.. code-block:: python\n\n    client.sudo_mode = mode  # where mode is True or False\n\nSSH Client supports sFTP for working with remote files:\n\n.. code-block:: python\n\n    with client.open(path, mode='r') as f:\n        ...\n\nFor fast remote paths checks available methods:\n\n- `exists(path)` -\u003e `bool`\n\n.. code-block:: python\n\n    \u003e\u003e\u003e conn.exists('/etc/passwd')\n    True\n\n- `stat(path)` -\u003e `paramiko.sftp_attr.SFTPAttributes`\n\n.. code-block:: python\n\n    \u003e\u003e\u003e conn.stat('/etc/passwd')\n    \u003cSFTPAttributes: [ size=1882 uid=0 gid=0 mode=0o100644 atime=1521618061 mtime=1449733241 ]\u003e\n    \u003e\u003e\u003e str(conn.stat('/etc/passwd'))\n    '-rw-r--r--   1 0        0            1882 10 Dec 2015  ?'\n\n- `isfile(path)` -\u003e `bool`\n\n.. code-block:: python\n\n    \u003e\u003e\u003e conn.isfile('/etc/passwd')\n    True\n\n- `isdir(path)` -\u003e `bool`\n\n.. code-block:: python\n\n    \u003e\u003e\u003e conn.isdir('/etc/passwd')\n    False\n\nAdditional (non-standard) helpers:\n\n- `mkdir(path: str)` - execute mkdir -p path\n- `rm_rf(path: str)` - execute rm -rf path\n- `upload(source: str, target: str)` - upload file or from source to target using sFTP.\n- `download(destination: str, target: str)` - download file from target to destination using sFTP.\n\nSubprocess specific\n-------------------\nKeyword arguments:\n\n- cwd - working directory.\n- env - environment variables dict.\n\n.. note:: `shell=true` is always set.\n\nasync_api.Subprocess specific\n-----------------------------\n\nAll standard methods are coroutines. Async context manager also available.\n\nExample:\n\n.. code-block:: python\n\n    async with helper:\n      result: ExecResult = await helper.execute(\n          command,  # type: str | Iterable[str]\n          verbose=False,  # type: bool\n          timeout=1 * 60 * 60,  # type: int | float | None\n          **kwargs\n      )\n\nTesting\n=======\nThe main test mechanism for the package `exec-helpers` is using `tox`.\nAvailable environments can be collected via `tox -l`\n\nCI systems\n==========\nFor code checking several CI systems is used in parallel:\n\n1. `GitHub actions: \u003chttps://github.com/python-useful-helpers/exec-helpers/actions\u003e`_ is used for checking: PEP8, pylint, bandit, installation possibility and unit tests.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpython-useful-helpers%2Fexec-helpers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpython-useful-helpers%2Fexec-helpers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpython-useful-helpers%2Fexec-helpers/lists"}