{"id":24567509,"url":"https://github.com/suprasummus/pgspawn","last_synced_at":"2025-07-26T23:40:08.245Z","repository":{"id":56350042,"uuid":"126625654","full_name":"SupraSummus/pgspawn","owner":"SupraSummus","description":"Spawn graph of processes that communicate with each other via UNIX pipes","archived":false,"fork":false,"pushed_at":"2020-11-12T21:58:00.000Z","size":168,"stargazers_count":1,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-24T12:49:08.797Z","etag":null,"topics":["descriptor","graph","pipeline","unix-command","unix-pipes","unix-socket"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SupraSummus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-24T17:55:09.000Z","updated_at":"2020-11-12T21:56:47.000Z","dependencies_parsed_at":"2022-08-15T17:10:10.629Z","dependency_job_id":null,"html_url":"https://github.com/SupraSummus/pgspawn","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SupraSummus%2Fpgspawn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SupraSummus%2Fpgspawn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SupraSummus%2Fpgspawn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SupraSummus%2Fpgspawn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SupraSummus","download_url":"https://codeload.github.com/SupraSummus/pgspawn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243971157,"owners_count":20376785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["descriptor","graph","pipeline","unix-command","unix-pipes","unix-socket"],"created_at":"2025-01-23T13:18:55.144Z","updated_at":"2025-03-17T03:41:34.449Z","avatar_url":"https://github.com/SupraSummus.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"PGSpawn\n=======\n\n[![Build Status](https://travis-ci.com/SupraSummus/pgspawn.svg?branch=master)](https://travis-ci.com/SupraSummus/pgspawn)\n\nDead simple utility for spawning graph of processes connected by streams in UNIX system. Each process is being equiped with specified UNIX pipes and sockets at specified file descriptors. Whole graph description is contained in a YAML file.\n\n`pgspawn` is awesome! Why? Here are some arguments for it:\n\n* It follows the UNIX philosophy.\n* It's simple and understandable.\n* It uses standard syntax - YAML is well known and pretty.\n* It uses standard pipe semantics - UNIX pipe is battle-tested.\n* It's language agnostic. `pgspawn` doesn't care about language - it spawns just *processes*.\n* It's efficient. After spawning phase all work is done by OS.\n\nInstall\n-------\n\nPackage is available from pypi\n\n    pip install pgspawn\n\nExamples\n--------\n\nCheck the `examples/` directory.\n\nAs input `pgspawn` takes YAML file with graph description in it.\n\n### id\n\nA very simple, single-node graph can be as follows:\n\n    $ cat examples/id.yml\n    nodes:\n      - command: [cat]\n    $ echo abc | pgspawn examples/id.yml\n    abc\n\nIt spawns `cat` program and doesn't do anything about file descriptors,\nso child process inherits standard fds (probably stdin, stdout, stderr).\n\n### yes\n\nWe can do more complex. Lets write `yes` program counterpart:\n\n    $ cat examples/yes.yml\n    nodes:\n      - command: [cat]\n        outputs:\n          1: feedback\n      - command: [tee, /proc/self/fd/3]\n        inputs:\n          0: feedback\n        outputs:\n          3: feedback\n    $ echo y | pgspawn examples/yes.yml\n    y\n    y\n    y\n    ...\n\nWhat it does is create pipe (named internally `feedback`) and use it to\nfeed output into input. Section `outputs: {1: feedback}` describes that\nfile descriptor 1 used by `cat` (it's stdout) is fed into our pipe.\nSection `inputs: {0: feedback}` denotes that fd 0 of `tee` program is\nread from `feedback` pipe.\n\nGraph drawn with explicitly connected stdin and stdout:\n\n![](images/yes_explicite.png)\n\n### shebang and pgspawn\n\nYAML allows for `#`-comments so they mix well together. Take a look at `examples/executable`:\n\n    #!/usr/bin/env pgspawn\n    nodes:\n     - command: [echo, 'hashbang!']\n\nAnd I can do this:\n\n    $ examples/executable\n    hashbang!\n\n### swap stdout-stderr\n\nIt's possible to use parent's program fds in `inputs` and `outputs` descriptions.\nJust give them names and roll:\n\n    $ cat examples/swap.yml\n    outputs:\n      stdout: 1\n      stderr: 2\n    nodes:\n      - command: [bash, -c, echo \"I'm stdout\"; echo \"I'm stderr\" \u003e\u00262;]\n        outputs:\n          1: stderr\n          2: stdout\n    $ pgspawn examples/swap.yml \u003e /dev/null\n    I'm stdout\n    $ pgspawn examples/swap.yml 2\u003e /dev/null\n    I'm stderr\n\nSimilar you can do with `inputs` (see `examples/id_explicite.yml`).\n\n![](images/swap.png)\n\n### server\n\nMore complicated example is shown in `examples/server.yml`.\nIt's a TCP chat with expression evaluation.\n\n![](images/server.png)\n\n### sockets\n\nSimple example with use of socket-connected processes is shown in `examples/socket.yml`.\n\n![](images/socket.png)\n\npg2dot\n------\n\nFor documentation purposes there is `pg2dot` program that converts YAML\ndescription of graph into [DOT file](https://en.wikipedia.org/wiki/DOT_(graph_description_language))\nsupported by [graphviz](http://www.graphviz.org/).\n\nTo generate image run something like:\n\n    cat examples/yes_explicite_full.yml | pg2dot | dot -T png -o graph.png\n\nNotes on concurrent reading and writing to fd\n---------------------------------------------\n\n`pgspawn` allows to make multiple programs write or read from single fd. If you do this you better be aware of what to expect.\n\nWhen multiple programs are writing into single pipe content gets interlaced, but there are rules. One can enforce write atomicity by writing small enough chunks. POSIX defines that size (PIPE_BUF) to be at least 512 bytes. (Yeah, citation needed.)\n\nFor concurrent reads matter is worse. There are some ways to make atomic read but all of them (no proof for that) rely on implementation, not on standard. There is hopeful [chapter in libc manual](https://www.gnu.org/software/libc/manual/html_node/Pipe-Atomicity.html) but `man 3 read` tells:\n\n\u003e The behavior of multiple concurrent reads on the same pipe, FIFO, or terminal device is unspecified.\n\nNotes on socket usage\n---------------------\n\nOf course socket can be modelled as a pair of unidirectional pipes. Such pipes will be connected at two fds in child process. Some programs may not support that and expect single fd. `pgspawn` can create pair of connected, anonymous UNIX domain sockets (like `socketpair()` from `\u003csys/socket.h\u003e`) and pass them to child processes. Such a connection can be shared only between two processes unlike unidirectional pipe that can be used by many more processes.\n\nRunning tests\n-------------\n\nThey are contained in `test.sh` file and exmaples directories.\n\nRunning is not so standard. I do it like this:\n\n    python setup.py develop --user\n    ./test.sh\n\nSee also\n--------\n\n* [pipexec](https://github.com/flonatel/pipexec) - similar tool\n* [dgsh](https://www2.dmst.aueb.gr/dds/sw/dgsh/) - extending UNIX pipeline to DAG\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuprasummus%2Fpgspawn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuprasummus%2Fpgspawn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuprasummus%2Fpgspawn/lists"}