{"id":34054886,"url":"https://github.com/osomdev/pyshrimp","last_synced_at":"2026-04-07T06:31:11.759Z","repository":{"id":62583445,"uuid":"415397002","full_name":"osomdev/pyshrimp","owner":"osomdev","description":"PyShrimp - combination of utilities designed to support easy creation of small python scripts.","archived":false,"fork":false,"pushed_at":"2024-05-02T15:02:18.000Z","size":555,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-15T20:59:23.200Z","etag":null,"topics":["python","scripting","shell"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/osomdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-10-09T18:58:39.000Z","updated_at":"2024-05-02T13:50:52.000Z","dependencies_parsed_at":"2024-03-10T16:38:19.060Z","dependency_job_id":null,"html_url":"https://github.com/osomdev/pyshrimp","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/osomdev/pyshrimp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osomdev%2Fpyshrimp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osomdev%2Fpyshrimp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osomdev%2Fpyshrimp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osomdev%2Fpyshrimp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/osomdev","download_url":"https://codeload.github.com/osomdev/pyshrimp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osomdev%2Fpyshrimp/sbom","scorecard":{"id":713787,"data":{"date":"2025-08-11","repo":{"name":"github.com/osomdev/pyshrimp","commit":"2a5e9d7cca789d96dae9a4aadd926da26ef49aee"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/run-tests.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/run-tests.yml:31: update your workflow using https://app.stepsecurity.io/secureworkflow/osomdev/pyshrimp/run-tests.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/run-tests.yml:34: update your workflow using https://app.stepsecurity.io/secureworkflow/osomdev/pyshrimp/run-tests.yml/master?enable=pin","Warn: pipCommand not pinned by hash: Dockerfile:10-12","Warn: pipCommand not pinned by hash: Dockerfile:10-12","Warn: pipCommand not pinned by hash: Dockerfile:17","Warn: pipCommand not pinned by hash: docker/post_release_test/Dockerfile:10","Warn: pipCommand not pinned by hash: docker/post_release_test/scripts/run-post-release-test.sh:10","Warn: pipCommand not pinned by hash: docker/post_release_test/scripts/run-post-release-test.sh:13","Warn: pipCommand not pinned by hash: .github/workflows/run-tests.yml:48","Warn: pipCommand not pinned by hash: .github/workflows/run-tests.yml:49","Warn: pipCommand not pinned by hash: .github/workflows/run-tests.yml:50","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   9 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-22T08:54:24.097Z","repository_id":62583445,"created_at":"2025-08-22T08:54:24.097Z","updated_at":"2025-08-22T08:54:24.097Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31503380,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","scripting","shell"],"created_at":"2025-12-14T02:31:06.776Z","updated_at":"2026-04-07T06:31:11.754Z","avatar_url":"https://github.com/osomdev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PyShrimp\n\n\u003cimg src=\"https://osomdev.github.io/pyshrimp/assets/img/logo/pyshrimp_logo_transparent_250.png\" align=\"right\" alt=\"PyShrimp logo\"\u003e\n\nPyShrimp is combination of utilities designed to support easy creation of small python scripts instead of getting your\nhands dirty with shell scripting languages like bash.\n\nWhen trying to write simple script to hack some ad-hoc task it's easy to end up with following choices:\n\n- use bash - it's easy to write simple script which invokes some processes. But then it's hard to do something more\n  advanced (lacking good support for types, lacking support for arrays...)\n- use python - is far better in handing different types, have good support for various collections, can easily be\n  expanded by installing dependencies from pip. But then there is painful overhead in setting up such script\n  (handling virtualenv to separate dependencies, or using one big env for all scripts) and doing more complex process\n  invocations beyond simple `subprocess.check_output(...)` is also far from ideal.\n\nThe PyShrimp aims at solving this - it's purpose is to remove barriers so developers can use python for simple\nshell-scripting purpose. No more over-grown bash scripts, no more pain with env setup or subprocess handling!\n\nNote: the features provided are not expected to replace poetry and other tools like that - if you can afford the\ncomplexity then it's probably better to use those ;). But for single-file scripts it should be way easier to go with\nPyShrimp.\n\n## TL;DR: Showcase\n\nThe rest of README explains features in detail. For those impatient there is quick example script - the code\nis often worth 1000 words (although it covers only small subset of features):\n\n```python\n#!/usr/bin/env pyshrimp\n# $opts: magic,devloop\n# $requires: click==7.0,requests==2.22.0\nimport requests\nfrom click import style\n\nfrom pyshrimp import log, as_dot_dict\n\n\ndef slog(message, **kwargs):\n    log(style(message, **kwargs))\n\n\nslog('Loading issues...')\n\nres = as_dot_dict(\n    requests.get(\n        'https://jira.atlassian.com/rest/api/2/search', params={\n            'jql': 'created \u003e -1d',\n            'maxResults': '5'\n        }\n    ).json(),\n    'res'\n)\n\nslog(f'Issues ({res.total} in total):', fg='green', bold=True)\n\nfor issue in res.issues:\n    line_color = 'red' if issue.fields.issuetype.name == 'Bug' else None\n    f = issue.fields\n    slog(f' * [{issue.key}] [{f.issuetype.name}] [{f.status.name}] {f.summary}', fg=line_color)\n```\n\nTo run this script you need to have the PyShrimp installed:\n\n```shell\npip install pyshrimp\nchmod +x thescript.py\n./thescript.py\n```\n\nPyShrimp will automatically initialize new virtual environment, install dependencies declared in file header,\nand execute the script with devloop:\n\n![Showcase output](https://osomdev.github.io/pyshrimp/assets/img/pyshrimp_showcase_output.png)\n\n## Features\n\n### Virtual env setup - `$requires`\n\nPyShrimp parses script header and looks for the `# $requires: ...` lines.\nEach such line can contain one or more pip-style requirements, for example:\n\n```python\n#!/usr/bin/env pyshrimp\n# $requires: click==7.0,requests==2.22.0\n# $requires: PyYAML==5.3.1\n```\n\nBefore script execution PyShrimp will create dedicated virtual env in `~/.cache/pyshrimp/{hash}/`\nwhere `{hash}` is hash value from the requirements. The environment is created only once - subsequent\nruns of script will use already existing environment. Scripts with exactly the same dependencies will\nre-use single virtual environment.\n\nThe `$requirements_file` can be used to load requirements list from external file:\n\n```python\n#!/usr/bin/env pyshrimp\n# $requirements_file: requirements.txt\n```\n\nIt is also possible to use alternative keywords: `$requirementsFile`, `$requires_file`, `$requiresFile`.\n\n### Creation of new script\n\nTo quickly create new script just run the `new` command:\n\n```shell\npyshrimp new my-new-script.py\n```\n\n[![](https://osomdev.github.io/pyshrimp/assets/img/asciinema_script_creation_snapshot.png)](\n  https://asciinema.org/a/9dDyBs1n1YkNtJBchFv3yZCt6?autoplay=1\n)\n\nThe file created will contain skeleton of script. Script will have the executable mode set already.\n\n### Script startup - `run`\n\nThe `run` function makes it easy to run the main function with some extra capabilities.\nThe default behavior is to set up logging and run the script main function.\nIn addition to this it's possible to enable additional tweaks:\n\n* `devloop` - runs script in loop - PyShrimp will re-execute script automatically after it changes\n* `elevate` - ensures that script is running as root - uses sudo to elevate permissions\n\nExample script with both features enabled:\n\n```python\n#!/usr/bin/env pyshrimp\nfrom pyshrimp import run, log\n\n\ndef main():\n  log('Hello world!')\n\n\nrun(main, devloop=True, elevate=True)\n```\n\n### Power of magic\n\nThere is alternative 'magic' way to run script setup (logging configuration) and achieve the bonus features\nlike devloop and elevate. This is called magic as the PyShrimp no longer will only set up virtualenv but also\nwill wrap the code with extra \"magical\" setup.\n\nThe following script will have exactly the same behavior as previously presented with `run` method:\n\n```python\n#!/usr/bin/env pyshrimp\n# $opts: magic,devloop,elevate\nfrom pyshrimp import log\n\nlog('Hello world!')\n```\n\nAs you can see there is less \"boilerplate\" code when using this approach.\n\nThere is however one downside - the code run with just `python` will behave differently - the magic options will not\nactivate and also the logging will be not configured. This can be surprising when running script with debugger so\nuse the magic wisely ;).\n\n### Useful utilities\n\nThere are few useful utilities provided:\n\n* `as_dot_dict` - creates dictionary wrapper with support for property-like access \n  to the values: `as_dot_dict(d).some_key.some_list[1].some_value`\n* `unwrap_dot_dict` - un-wraps DotDicts back into the raw dict/list\n* `ls`, `glob_ls` - lists files and directories\n* `write_to_file`, `read_file`, `read_file_bin` - file content manipulation\n* `chmod_set`, `chmod_unset` - sets/unsets file mode bits\n* `acquire_file_lock`, `FileBasedLock` - handles file based locking\n* `re_match_all` - runs regular expression matching across the list and returns selected \n  group from the matched elements\n* `in_background` - runs function in background thread pool\n* [`StringWrapper`](src/pyshrimp/utils/string_wrapper.py) - provides few methods especially useful for parsing process output\n* `parse_table` - parses table-like output into `ParsedTable`\n* `create_regex_splitter` - creates `regex_splitter` - useful to handle unusual table/column-like output\n* `wait_until`, `wait_until_gen` - handles waiting for some result with timeout using periodic polling\n\nYou can see example usage in [examples](examples) and also in [tests](tests).\n\n### Easily run programs - simple subprocess helper\n\nThe subprocess helper simplifies the task of running processes and handling the results.\n\n```python\nfrom pyshrimp import run_process\nprint(\n  run_process('echo -n 123 | wc -c', run_in_shell=True).raise_if_not_ok().standard_output\n)\n```\n\n### More advanced subprocess helper\n\nThe `cmd` and `shell_cmd` will produce `Command` object which can be executed with extra params:\n\n```python\nfrom pyshrimp import shell_cmd\nwc_c = shell_cmd('echo -n \"$1\" | wc -c', check=True)\nprint(wc_c('123456789').standard_output.strip()) // 9\nprint(wc_c.exec('123').standard_output.strip()) // 3\n```\n\nIt is worth noting here that standard_output and error_output\nare [wrapped with `StringWrapper`](src/pyshrimp/utils/string_wrapper.py) to ease output parsing.\n\n## Pipeline support\n\n### Motivation\n\nThe nice property about bash scripts is how easy it is to executed process and pass the output between processes. It's\nall possible in python but the overhead is really too big to use in small scripts. It's more convenient to fallback into\nsubprocess with partial shell script than glue together the processes directly in python code.\n\n### Solution\n\nThe `ExecutionPipeline` is designed to address those concerns. You can easily stitch together few processes and\nfunctions together to process the input and output just like in shell scripts.\n\nThere are two approaches - standard object-oriented python code and more shell like pipeline syntax.\n\n### Object oriented usage\n\n```python\nfrom pyshrimp import ExecutionPipeline, cmd\np = ExecutionPipeline()\n# feed pipeline with text input\np.attach_text('Hello world!')\n# run the wc command\np.attach(cmd('wc'))\n# run awk, using the shell wrapper\np.attach(\"awk '{print $3}'\")\n# process the output with function - pad with zeros\np.attach_function(\n    lambda stdin: f'{int(stdin.strip()):05}'\n)\n# close pipeline and collect output\nres = p.close().stdout\nprint(res) # 00012\n```\n\n### Shell-like pipe syntax\n\nThe shell-like syntax is \"abusing\" the python feature which allows overriding behavior of binary or operator. It's not\nvery elegant (could be confusing for people) but on the other hand this fits nicely the purpose - being easy replacement\nof shell pipes.\n\n```python\nfrom pyshrimp import PIPE, PIPE_END_STDOUT, cmd\nres = (\n    # feed pipeline with text input\n    PIPE.text('Hello world!')\n    # run the wc command\n    | cmd('wc')\n    # run awk, using the shell wrapper\n    | \"awk '{print $3}'\"\n    # process the output with function - pad with zeros\n    | (lambda stdin: f'{int(stdin.strip()):05}')\n    # close pipeline and collect output\n    | PIPE_END_STDOUT\n)\nprint(res) # 00012\n```\n\n### Mixed approach\n\nYou can mix the shell-like and object-oriented syntax (if you have good reason to do so ;)). Under the cover\nthe `ExecutionPipeline` is being used even in the shell-like syntax.\n\n```python\nfrom pyshrimp import PIPE, cmd\np = (\n    # feed pipeline with text input\n    PIPE.text('Hello world!')\n    # run the wc command\n    | cmd('wc')\n    # run awk, using the shell wrapper\n    | \"awk '{print $3}'\"\n)\n# process the output with function - pad with zeros\np.attach_function(lambda stdin: f'{int(stdin.strip()):05}')\n# close pipeline and collect output\nres = p.close().stdout\nprint(res) # 00012\n```\n\n### Limitations\n\nThings obviously missing in current version that you should be aware of:\n\n- Limited error handling support\n- Only text streams are officially supported, with UTF-8 hardcoded\n- Only connection of standard output is supported - error output behavior is undetermined (most likely flows to stderr)\n\nThose limitations can be addressed in future (if there is enough demand and willingness to introduce the change).\n\n## Troubleshooting\n\n### Diagnosing virtual env setup failure\n\nYou can set `PYSHRIMP_LOG` environment variable to `1` this will instruct boostrap code to\nproduce diagnostic messages:\n\n```shell\n% PYSHRIMP_LOG=1 ./show_recently_created_issues.py a b\n[PyShrimp:bootstrap] INFO: target: ./show_recently_created_issues.py\n[PyShrimp:bootstrap] INFO: args: ['a', 'b']\n[PyShrimp:bootstrap] INFO: Using requirements d82e6efbb5ea5ba895b6fe103b4c50bf3ac75eb3: [\n  'pyshrimp', 'click==7.0'\n]\n[PyShrimp:bootstrap] INFO: Executing the script: [\n  '~/.cache/pyshrimp/virtual_envs/d82e6e.../bin/python', '-u', \n  '-m', 'pyshrimp._internal.wrapper.magicwrapper', \n  '--', './show_recently_created_issues.py', 'a', 'b'\n]\n  \nHello world\n```\n\n## Supported environment\n\nThis project was developed on **Ubuntu Linux**. It should work with any linux system, but I can imagine the tests\nfailing in case some system binaries are missing.\n\nThe project is also tested on macOS before new version is released.\n\nSome parts for sure will not work on Windows (e.g. shell wrapping is depending on bash).\nThe author does not have plans to introduce support for Windows but contributions are welcome ;).\n\n## License\n\nThe project is licensed under [MIT License](./LICENSE.txt) with exceptions listed below.\n\nProject license exceptions:\n\n1. The files in [doc/assets/img/logo](doc/assets/img/logo) and [docs/assets/img/logo](doc/assets/img/logo) directory\n   are licensed under [CC BY-SA 3.0 license](https://creativecommons.org/licenses/by/3.0/legalcode).\n\n## Contributions\n\nFeel free to contribute to this project - I'll do my best to review and accept contributions.\n\nPlease include at least some happy-path tests for your changes.\n\n## Imaginary Q\u0026A\n\nQ: Shouldn't this project be separated into few ones (e.g. pipelines, commands, script bootstrap)?  \nA: Probably yes. And maybe it will be split in the future. But for now it's more convenient to manage single project.\n\n## Credits\n\n- The logo was created using [\"Shrimp\" icon](https://thenounproject.com/elabans/collection/seafood/?i=541402)\n  created by [\"elmars\"](https://thenounproject.com/elabans/) and published \n  under [CC BY-SA 3.0 license](https://creativecommons.org/licenses/by/3.0/us/legalcode).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosomdev%2Fpyshrimp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosomdev%2Fpyshrimp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosomdev%2Fpyshrimp/lists"}