{"id":24374549,"url":"https://github.com/hansalemaos/cynonblockingsubprocess","last_synced_at":"2026-02-22T21:44:38.723Z","repository":{"id":273026141,"uuid":"918505915","full_name":"hansalemaos/cynonblockingsubprocess","owner":"hansalemaos","description":"Nogil subprocess for Python with stdout/stderr capturing and stdin writing - without deadlocking!","archived":false,"fork":false,"pushed_at":"2025-01-18T05:19:25.000Z","size":12,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-15T21:15:32.994Z","etag":null,"topics":["capturing","cpp","fast","nogil","nonblocking","popen","printing","stderr","stdin","stdout","subprocess"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/cynonblockingsubprocess/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hansalemaos.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-18T05:19:23.000Z","updated_at":"2025-04-08T16:33:55.000Z","dependencies_parsed_at":"2025-01-18T05:31:50.522Z","dependency_job_id":"d9ea3b1a-5f49-44b1-a399-aa66d91873a7","html_url":"https://github.com/hansalemaos/cynonblockingsubprocess","commit_stats":null,"previous_names":["hansalemaos/cynonblockingsubprocess"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fcynonblockingsubprocess","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fcynonblockingsubprocess/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fcynonblockingsubprocess/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hansalemaos%2Fcynonblockingsubprocess/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hansalemaos","download_url":"https://codeload.github.com/hansalemaos/cynonblockingsubprocess/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249153950,"owners_count":21221330,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["capturing","cpp","fast","nogil","nonblocking","popen","printing","stderr","stdin","stdout","subprocess"],"created_at":"2025-01-19T05:40:39.107Z","updated_at":"2026-02-22T21:44:33.691Z","avatar_url":"https://github.com/hansalemaos.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nogil subprocess for Python with stdout/stderr capturing and stdin writing - without deadlocking!\n\n## What is it?\n\n**cynonblockingsubprocess** is a Cython-based wrapper around a C++ class called `ShellProcessManager`, defined in `nonblockingsubprocess.hpp`. Its purpose is to start and manage a shell process (e.g., `/bin/bash`, `\"C:\\\\Windows\\\\System32\\\\cmd.exe\"`, or any other command-line application), allowing you to:\n\n- Write commands to the shell’s standard input.\n- Read from the shell’s standard output and standard error without risking deadlocks.\n- Run two background threads to non-blockingly capture `stdout` and `stderr` in **nogil** mode (reducing Python GIL contention).\n- Optionally print `stdout` and/or `stderr` in real time while still retaining the captured output.\n\n## Features\n\n- **Non-blocking I/O**: Spawns separate threads to read `stdout` and `stderr` in the background, preventing the parent process from freezing or deadlocking.\n- **Configurable Buffers**: Control the maximum length of captured `stdout` and `stderr` buffers before data is truncated, similar to a ring buffer (or `collections.deque`).\n- **OS Compatibility**:\n  - On **Windows**, you can customize process creation flags and environment parameters.\n  - On **Linux/Unix**, the Windows-specific parameters are ignored, but the shell process is still managed the same way.\n- **Graceful Shutdown**: Call `stop_shell()` to terminate the subprocess and background threads safely. When using subprocess.Popen, shells might keep open in the background, even after calling .kill() or .terminate(). This should not happen here! The C++ destructor takes care of closing the shell.\n\n## Installation and Requirements\n\n### `pip install cynonblockingsubprocess`\n\n- **Cython** (to compile the `.pyx` / `.pxd` file).\n- A **C++ compiler** (minimum version 11) compatible with your platform (e.g., `MSVC` on Windows or `g++`/`clang++` on Linux).\n- The code will compile the first time you import it\n\n\n## Usage example Python\n\n```py\nfrom cynonblockingsubprocess import CySubProc\nfrom time import sleep\nfrom platform import platform\n\niswindows = \"win\" in platform().lower()\n\n\n# shell_command (bytes or str): Path or command to start (e.g., \"C:\\\\Windows\\\\System32\\\\cmd.exe\" or \"/bin/bash\").\n# buffer_size (int): Size of the internal buffer for reading output (default 4096).\n# stdout_max_len (int): Maximum amount of data retained in the stdout buffer (default 4096).\n# stderr_max_len (int): Maximum amount of data retained in the stderr buffer (default 4096).\n# exit_command (bytes or str): Command used internally to gracefully stop the shell (default b\"exit\").\n# print_stdout (bool): If True, prints all stdout in real time to the console (default False).\n# print_stderr (bool): If True, prints all stderr in real time to the console (default False).\n\ntete = CySubProc(\n    shell_command=\"C:\\\\Windows\\\\System32\\\\cmd.exe\" if iswindows else \"/bin/bash\", # cross plattform\n    buffer_size=4096,\n    stdout_max_len=4096,\n    stderr_max_len=4096,\n    exit_command=b\"exit\",\n    print_stdout=False,\n    print_stderr=False,\n)\n\n# Start the shell process with specific creation flags and environment parameters.\n# On Posix, all arguments are ignored!\n\n# Detailed information can be found on Microsoft's website https://learn.microsoft.com/en-us/windows/win32/procthread/process-creation-flags\n\n# Parameters\n# ----------\n# creationFlag : int, optional\n#     A custom flag for process creation, by default 0.\n# creationFlags : int, optional\n#     Additional flags controlling process creation, by default 0x08000000.\n# wShowWindow : int, optional\n#     Flags controlling how the window is shown (e.g., hidden, normal, minimized),\n#     by default 1 (SW_SHOWNORMAL).\n# lpReserved : bytes or str, optional\n#     Reserved parameter for process creation, by default None.\n# lpDesktop : bytes or str, optional\n#     The name of the desktop for the process, by default None.\n# lpTitle : bytes or str, optional\n#     The title for the new console window, by default None.\n# dwX : int, optional\n#     X-coordinate for the upper-left corner of the window, by default 0.\n# dwY : int, optional\n#     Y-coordinate for the upper-left corner of the window, by default 0.\n# dwXSize : int, optional\n#     Width of the window, by default 0.\n# dwYSize : int, optional\n#     Height of the window, by default 0.\n# dwXCountChars : int, optional\n#     Screen buffer width in character columns, by default 0.\n# dwYCountChars : int, optional\n#     Screen buffer height in character rows, by default 0.\n# dwFillAttribute : int, optional\n#     Initial text and background colors if used in a console, by default 0.\n# dwFlags : int, optional\n#     Flags that control how the creationFlags are used, by default 0.\n# cbReserved2 : int, optional\n#     Reserved for C runtime initialization, by default 0.\n# lpReserved2 : bytes or str, optional\n#     Reserved for C runtime initialization, by default None.\n\ntete.start_shell() # This function must be always called first, if not, you probably will get segmentation faults!\n\n\n# Write a command or input data to the shell process's stdin.\n\n# Parameters\n# ----------\n# cmd : bytes or str\n#     The command or data to send to the process via stdin.\ntete.stdin_write(\"ls -l\") # Writing an existing command \nsleep(1) # Wait a little for the output\n\n\n# Retrieve the current contents of the shell's standard output as bytes, and clears the C++ vector\n\n# Returns\n# -------\n# bytes\n#     The raw bytes from the shell's stdout.\nprint(tete.get_stdout().decode()) # there will be something here\n\n\n# Retrieve the current contents of the shell's standard error as bytes, and clears the C++ vector.\n\n# Returns\n# -------\n# bytes\n#     The raw bytes from the shell's stderr.\nprint(tete.get_stderr().decode()) # will be empty\ntete.stdin_write(\"lxs xx-lxxxx\") # command does not exist\nsleep(1)\nprint(tete.get_stdout())  # will be empty\nprint(tete.get_stderr()) # there will be something here\ndel tete # closes the shell automatically! If you want, you can also call proc.stop_shell()\n\n```\n\n## Usage example C++ (stack)\n\n```cpp\n#include \"nonblockingsubprocess.hpp\"\n\nint main(int argc, char *argv[])\n{\n    while (true)\n    {\n#ifdef _WIN32\n        std::string shellcmd = \"C:\\\\Windows\\\\System32\\\\cmd.exe\";\n#else\n        std::string shellcmd = \"/bin/bash\";\n#endif\n        // arguments: std::string shell_command, size_t buffer_size = 4096, size_t stdout_max_len = 4096, size_t stderr_max_len = 4096, std::string exit_command = \"exit\", int print_stdout = 1, int print_stderr = 1\n        ShellProcessManager proc{shellcmd, 4096, 4096, 4096, \"exit\", 1, 1};\n        bool resultproc = proc.start_shell(); //optional arguments for Windows: DWORD creationFlag = 0, DWORD creationFlags = CREATE_NO_WINDOW, WORD wShowWindow = SW_NORMAL, LPSTR lpReserved = nullptr, LPSTR lpDesktop = nullptr, LPSTR lpTitle = nullptr, DWORD dwX = 0, DWORD dwY = 0, DWORD dwXSize = 0, DWORD dwYSize = 0, DWORD dwXCountChars = 0, DWORD dwYCountChars = 0, DWORD dwFillAttribute = 0, DWORD dwFlags = 0, WORD cbReserved2 = 0, LPBYTE lpReserved2 = nullptr\n        std::cout \u003c\u003c \"resultproc: \" \u003c\u003c resultproc \u003c\u003c std::endl;\n        proc.stdin_write(\"ls -l\");\n        sleepcp(100);\n        auto val = proc.get_stdout();\n        std::cout \u003c\u003c \"stdout: \" \u003c\u003c val \u003c\u003c std::endl;\n        sleepcp(100);\n        proc.stdin_write(\"ls -l\");\n        sleepcp(100);\n        auto val2 = proc.get_stdout();\n        std::cout \u003c\u003c \"stderr: \" \u003c\u003c val \u003c\u003c std::endl;\n        proc.stop_shell(); // optional: automatically called by the destructor\n        sleepcp(1000);\n    }\n}\n```\n\n## Usage example C++ (heap)\n\n```cpp\n#include \"nonblockingsubprocess.hpp\"\n\nint main(int argc, char *argv[])\n{\n    while (true)\n    {\n#ifdef _WIN32\n        std::string shellcmd = \"C:\\\\Windows\\\\System32\\\\cmd.exe\";\n#else\n        std::string shellcmd = \"/bin/bash\";\n#endif\n\n        ShellProcessManager *proc = new ShellProcessManager{shellcmd, 4096, 4096, 4096, \"exit\", 1, 1};\n        bool resultproc = proc-\u003estart_shell();\n        std::cout \u003c\u003c \"resultproc: \" \u003c\u003c resultproc \u003c\u003c std::endl;\n        proc-\u003estdin_write(\"ls -l\");\n        sleepcp(100);\n        auto val = proc-\u003eget_stdout();\n        std::cout \u003c\u003c \"v1111111111: \" \u003c\u003c val \u003c\u003c std::endl;\n        sleepcp(100);\n        proc-\u003estdin_write(\"ls -l\");\n        sleepcp(100);\n        auto val2 = proc-\u003eget_stdout();\n        std::cout \u003c\u003c \"v2222222222: \" \u003c\u003c val \u003c\u003c std::endl;\n        proc-\u003estop_shell();\n        sleepcp(1000);\n    }\n    delete proc\n    std::cin.get();\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhansalemaos%2Fcynonblockingsubprocess","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhansalemaos%2Fcynonblockingsubprocess","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhansalemaos%2Fcynonblockingsubprocess/lists"}