{"id":13638452,"url":"https://github.com/anki-code/xontrib-pipeliner","last_synced_at":"2025-08-22T08:31:38.131Z","repository":{"id":46944212,"uuid":"260732533","full_name":"anki-code/xontrib-pipeliner","owner":"anki-code","description":"Let your pipe lines flow thru the Python code in xonsh.","archived":false,"fork":false,"pushed_at":"2024-06-07T09:27:50.000Z","size":153,"stargazers_count":56,"open_issues_count":4,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-07T18:11:46.633Z","etag":null,"topics":["data-engineering","data-engineering-pipeline","pipe","pipeline","pipelines","python","shell","xonsh","xontrib"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anki-code.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"custom":["https://github.com/anki-code","https://www.buymeacoffee.com/xxh","https://github.com/xonsh/xonsh#the-xonsh-shell-community"]}},"created_at":"2020-05-02T16:51:16.000Z","updated_at":"2024-06-07T09:27:53.000Z","dependencies_parsed_at":"2024-01-29T10:09:27.175Z","dependency_job_id":"70ea6738-faf8-4d08-ac11-6e5bb593f9e7","html_url":"https://github.com/anki-code/xontrib-pipeliner","commit_stats":{"total_commits":89,"total_committers":4,"mean_commits":22.25,"dds":0.1573033707865169,"last_synced_commit":"9fc6287212efdd3ef8fd94ead46d1e16bad2eab1"},"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anki-code%2Fxontrib-pipeliner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anki-code%2Fxontrib-pipeliner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anki-code%2Fxontrib-pipeliner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anki-code%2Fxontrib-pipeliner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anki-code","download_url":"https://codeload.github.com/anki-code/xontrib-pipeliner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230575851,"owners_count":18247484,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-engineering-pipeline","pipe","pipeline","pipelines","python","shell","xonsh","xontrib"],"created_at":"2024-08-02T01:00:46.111Z","updated_at":"2024-12-20T11:08:05.238Z","avatar_url":"https://github.com/anki-code.png","language":"Python","funding_links":["https://github.com/anki-code","https://www.buymeacoffee.com/xxh","https://github.com/xonsh/xonsh#the-xonsh-shell-community"],"categories":["Plugins"],"sub_categories":["Prompt tweaks"],"readme":"\u003cp align=\"center\"\u003e  \nEasily process the lines using pipes in \u003ca href=\"https://xon.sh\"\u003exonsh shell\u003c/a\u003e. Multicore processing supported.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e  \nIf you like the idea of pipeliner click ⭐ on the repo and \u003ca href=\"https://twitter.com/intent/tweet?text=Nice%20xontrib%20for%20the%20xonsh%20shell!\u0026url=https://github.com/anki-code/xontrib-pipeliner\" target=\"_blank\"\u003etweet\u003c/a\u003e.\n\u003c/p\u003e\n\n\n## Install\n```bash\nxpip install -U xontrib-pipeliner\necho 'xontrib load pipeliner' \u003e\u003e ~/.xonshrc\n# Reload xonsh\n```\n\n## Usage\nLet your pipe lines flow thru the Python code:\n```bash\n\u003ccmd\u003e | \u003c...\u003e | pl \"\u003cpreset name or lambda expression\u003e\" | \u003ccmd\u003e | \u003c...\u003e\n```\n\nExperimental:\n\n* `ppl` is to run multicore `pl`. It tested mostly on Linux. See \"Known issues\" section.\n* `plx` is the shorter way to execute the commands with pipe lines i.e. `ls /home | plx 'du -sh /home/{line}'`.\n\n## Examples\n\n### Presets\n\n```xsh\npl  # list of presets\n\necho \"  1\" | pl strip\n# 1\n\necho \"1,2,3\" | pl split ,\n['1', '2', '3']\n\necho \"a,b,c\" | pl split , | pl fromlist 0\n# a\n\necho xonsh pids is $(ps ax | grep xonsh | grep -v grep | pl split ' ' | pl fromlist 0)\n# xonsh pids is 56486 56913 56489\n\n```\n\nYou can set your own presets:\n```xsh\n$XONTRIB_PIPELINER_PRESETS = {\n    \"upper\": \"line.upper()\",\n    \"repeat\": lambda line, num, args: line * int(args[0])\n}\n\necho 'hello' | pl upper\n# HELLO\n\necho 'hey \\nhi ' | pl repeat 3\n# hey hey hey\n# hi hi hi\n```\n\n### Lambda string\n\nThere are two variables available in lambda expression:\n* `line` from pipe.\n* `num` of the line starts with 0.\n\n#### Python way to line modification\n```bash\nls -1 / | pl \"line + ' is here'\" | head -n 3\n```\n```\nbin is here\nboot is here\ndev is here\n```\n\n#### Line number\n```bash\nls -1 / | head -n 4 | pl \"f'{num} {line}'\"\n```\n```\n0 bin\n1 boot\n2 cdrom\n3 dev\n```\n\n#### Ignore line\n```bash\n$ ls -1 / | head -n 4 | pl \"f'{num} {line}' if num%2 == 0 else None\"\n```\n```\n0 bin\n2 cdrom\n```\n\n#### Splitting\n```bash\ncat /etc/passwd | head -n 3 | pl \"line.split(':')[6]\"\n```\n```\n/bin/bash\n/usr/sbin/nologin\n/usr/sbin/nologin\n```\n\n#### Imports\n```bash\nimport re\ncat /etc/passwd | head -n 3 | pl \"re.sub('/bin/bash', '/usr/bin/xonsh', line)\"\n```\n```\nroot:x:0:0:root:/root:/usr/bin/xonsh\ndaemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin\nbin:x:2:2:bin:/bin:/usr/sbin/nologin\n```\n\n#### Arrays\n```bash\ncat /etc/passwd | head -n 3 | pl \"line.split(':')\" | grep nologin | pl \"':'.join(eval(line)[::-1])\"\n```\n```\n/usr/sbin/nologin:/usr/sbin:daemon:1:1:x:daemon\n/usr/sbin/nologin:/bin:bin:2:2:x:bin\n```\n\n#### Python head\n```bash\npl \"'\\\\n'.join(list('ABCDEFG'))\" | pl \"line + ('!' if num%2 else '?')\" | grep '!'\n```\n```\nB!\nD!\nF!\n```\n\n#### Variables and operations chaining\nExpression is a lambda function so using variables and operations chaining since Python 3.8+ are available by trick with the walrus operator and the list:\n```bash\nls -1 / | head -n3 | pl \"[s:='b', line.replace(s, s.upper()+')')][-1]\"\n```\n```\nB)in\nB)oot\ndev\n```\n\n#### Execute command with the line\n```bash\nls / | head -n 3 | pl \"execx('du -sh /'+line) or 'Done command with /'+line\"\n```\n```\n0       /bin\nDone command with /bin\n840M    /boot\nDone command with /boot\n4,0K    /cdrom\nDone command with /cdrom\n```\nNote! If you do the operations with files (i.e. `pl \"execx(f'mv {line} prefix-{line}')\"`) you could catch `TypeError: an integer is required` error that relates to wrong access rights to files. Fix it with `chmod` and `chown` before pipelining.\n\n\n## Wrap pipeliner to get your own magic\n```python\naliases['my_lovely_pl'] = lambda a,i,o: aliases['pl']([\"'My lovely ' + \"+a[0]], i, o)\naliases['my_parallel_ppl'] = lambda a,i,o: aliases['ppl']([\"'My parallel ' + \"+a[0]], i, o)\n```\n```bash\nls / | head -n 3 | my_lovely_pl \"line + '!'\"\n# My lovely bin!\n# My lovely boot!\n# My lovely cdrom!\n\nls / | head -n 3 | my_parallel_ppl \"line + '!'\"\n# My parallel boot!\n# My parallel cdrom!\n# My parallel bin!\n```\nAdd your most useful solutions to xontrib-pipeliner. PRs are welcome!\n\n## Experimental\n\n### Syntax highlighting using xonsh prompt\n\nIf you're using xonsh prompt and want to use pipeliner with syntax highlighting instead of string there is experimental \nfeature that catch `pl @(\u003cpython\u003e)` calls and uses the expression from the xonsh python substitution as pipeliner argument.\nExample:\n\n```bash\necho echo | pl @(line + '!')\n# In the xonsh prompt it's equals to:\necho echo | pl \"line + '!'\" \n```\n\n### Syntax highlighting using xonsh macros\nTo avoid writing Python inside the string and get the syntax highlighting there is a tricky way with using [xonsh macro](https://xon.sh/tutorial_macros.html):\n```python\ndef py(code):\n    return code\n\necho 123 | pl @(py!(line + '2'))\n```\n\n### Multicore pipelining\nBy default pipeliner works using one CPU core. To use them all in parallel try `ppl` command:\n```bash\nhead /etc/passwd | ppl \"str(num) + ' ' + line.split(':')[0]\"\n```\n```\n1 daemon\n0 root\n2 bin\n4 sync\n5 games\n8 mail\n9 news\n6 man\n7 lp\n3 sys\n```\nNote! The order of result lines is unpredictable because lines will be processed in parallel. \nThe `num` variable contains the real line number. \n\n### Pipeliner exec\nThere are `plx` and `pplx` commands to run `execx(f\"{plx_command}\")` most shorter way.\n\nFor example when you want to rename files you can do it Pythonic way:\n```bash\nmkdir -p /tmp/plx-test \u0026\u0026 cd /tmp/plx-test\ntouch 111 222 333 \u0026\u0026 ls\n# 111 222 333\n\nls | plx \"mv {line} prefix-{line}\"\n# mv 111 prefix-111\n# mv 222 prefix-222\n# mv 333 prefix-333\n\nls\n# prefix-111 prefix-222 prefix-333\n```\nEcho example:\n```bash\nls | plx 'echo {line} # {num}'\n# echo prefix-111 # 0\n# prefix-111\n# echo prefix-222 # 1\n# prefix-222\n# echo prefix-333 # 2\n# prefix-333\n```\n\n### Pipeliner in xsh scripts\nBy default xsh scripts haven't rc-file with xontribs loading. To add pipeliner to your script just do `xontrib load pipeliner` before usage.\n\n## Known issues in experimental functions\n\n### plx: \"Bad file descriptor\" on huge amount of lines\n\nhttps://github.com/xonsh/xonsh/issues/4224\n\n### ppl: [On MacOS global variables are not accessible from child processes](https://bugs.python.org/issue39931) in multicore pipelining\n\nOn Mac you can't access to the xonsh context (global variables and functions) in the expression. PR is welcome!\n\n### ppl: On MacOS multicore pipelining freezes on end\n\nWorkaround is to add `cat` at the end: `echo 1 | ppl 'line' | cat`. PR is welcome!\n\n## Future\n\nPipeliner should be a part of xonsh and has shortcut and syntax highlighting. For example:\n```python\necho 'Pipeliner should be ' | pl @{line + 'a part of xonsh!'}\n# or\necho 'Pipeliner should be ' | ~(line + 'a part of xonsh!')\n```\n```\nPipeliner should be a part of xonsh!\n```\n\nIf you want to support this in xonsh add your Like and support message to [Python code substitution in subproc mode](https://github.com/xonsh/xonsh/issues/3945).\n\n## Links \n* This package is the part of [rc-awesome](https://github.com/anki-code/xontrib-rc-awesome) - awesome snippets of code for xonshrc in xonsh shell.\n* This package is the part of [ergopack](https://github.com/anki-code/xontrib-ergopack) - the pack of ergonomic xontribs.\n* This package was created with [xontrib cookiecutter template](https://github.com/xonsh/xontrib-cookiecutter).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanki-code%2Fxontrib-pipeliner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanki-code%2Fxontrib-pipeliner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanki-code%2Fxontrib-pipeliner/lists"}