{"id":13936446,"url":"https://github.com/shmuelamar/cbox","last_synced_at":"2025-04-09T19:18:57.620Z","repository":{"id":62561103,"uuid":"101936984","full_name":"shmuelamar/cbox","owner":"shmuelamar","description":"convert any python function to unix-style command","archived":false,"fork":false,"pushed_at":"2023-06-08T08:54:47.000Z","size":55,"stargazers_count":158,"open_issues_count":1,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-09T19:18:53.299Z","etag":null,"topics":["cli","command","pipes","python","python3","shell","shellscript","unix"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shmuelamar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-30T23:09:52.000Z","updated_at":"2024-04-17T22:27:06.000Z","dependencies_parsed_at":"2024-10-16T07:41:04.811Z","dependency_job_id":null,"html_url":"https://github.com/shmuelamar/cbox","commit_stats":{"total_commits":18,"total_committers":2,"mean_commits":9.0,"dds":"0.11111111111111116","last_synced_commit":"2d0cda5b3f61a55e530251430bf3d460dcd3732e"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shmuelamar%2Fcbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shmuelamar%2Fcbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shmuelamar%2Fcbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shmuelamar%2Fcbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shmuelamar","download_url":"https://codeload.github.com/shmuelamar/cbox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248094990,"owners_count":21046770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","command","pipes","python","python3","shell","shellscript","unix"],"created_at":"2024-08-07T23:02:40.814Z","updated_at":"2025-04-09T19:18:57.599Z","avatar_url":"https://github.com/shmuelamar.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# CBOX - CLI ToolBox\n\n[![PyPI](https://img.shields.io/pypi/v/cbox.svg)](https://pypi.python.org/pypi/cbox/0.1.0)\n[![PyPI](https://img.shields.io/pypi/pyversions/cbox.svg)](https://pypi.python.org/pypi/cbox/0.1.0)\n[![Build Status](https://travis-ci.org/shmuelamar/cbox.svg?branch=master)](https://travis-ci.org/shmuelamar/cbox)\n[![AppVeyor](https://img.shields.io/appveyor/ci/gruntjs/grunt.svg)](https://ci.appveyor.com/project/shmuelamar/cbox)\n[![Codecov](https://img.shields.io/codecov/c/github/shmuelamar/cbox.svg)](https://codecov.io/gh/shmuelamar/cbox)\n[![PyPI](https://img.shields.io/pypi/wheel/cbox.svg)]()\n[![PyPI](https://img.shields.io/pypi/l/cbox.svg)]()\n\n### convert any python function to unix-style command\n\nThe Unix Philosophy (from [wikipedia](https://en.wikipedia.org/wiki/Unix_philosophy#Origin)):\n\u003e    * *Write programs that do one thing and do it well.*\n\u003e    * *Write programs to work together.*\n\u003e    * *Write programs to handle text streams, because that is a universal interface.*\n\n\u003cbr /\u003e\n\n## Features\n* supports pipes\n* concurrency (threading or asyncio)\n* supports error handling (redirected to stderr)\n* supports for inline code in cli style\n* various output processing options (filtering, early stopping..)\n* supports multiple types of pipe processing (lines, chars..)\n* automatic docstring parsing for description and arguments help\n* automatic type annotation and defaults parsing\n* returns the correct exitcode based on errors\n* supports only python3 (yes this is a feature)\n* supports subcommands\n\n## Quickstart\n\n**install:**\n\n```bash\npip install -U cbox\n```\n\n**example usage:**\n```python\n#!/usr/bin/env python3\n# hello.py\nimport cbox\n\n@cbox.cmd\ndef hello(name: str):\n    \"\"\"greets a person by its name.\n\n    :param name: the name of the person\n    \"\"\"\n    print(f'hello {name}!')\n\nif __name__ == '__main__':\n    cbox.main(hello)\n```\n\n**run it:**\n\n```bash\n$ ./hello.py --name world\nhello world!\n\n$ ./hello.py --help\nusage: hello.py [-h] --name NAME\n\ngreets a person by its name.\n\noptional arguments:\n  -h, --help   show this help message and exit\n  --name NAME  the name of the person\n```\n\n**cli inline example:**\n\n```bash\n$ echo -e \"192.168.1.1\\n192.168.2.3\\ngoogle.com\" | cbox --modules re 're.findall(\"(?:\\d+\\.)+\\d+\", s)'\n192.168.1.1\n192.168.2.3\n```\n\n*for more info about cbox inline run `cbox --help`*\n\n\n## The Story\nonce upon a time, a python programmer named dave, had a simple text file. \n\n**langs.txt**\n```text\npython http://python.org\nlisp http://lisp-lang.org\nruby http://ruby-lang.org\n```\n\nall dave wanted is to get the list of languages from that file.\n\nour dave heard that unix commands are the best, so he started googling them out.\n\nhe started reading about *awk*, *grep*, *sed*, *tr*, *cut* and others but couldn't \nremember how to use all of them - after all he is a python programmer and wants to use python.\n\nfortunately, our little dave found out about **`cbox`** - a simple way to convert \nany python function into unix-style command line!\n\nnow dave can process files using python easily!\n\n### simple example\n```python\n#!/usr/bin/env python3\n# first.py\nimport cbox\n\n@cbox.stream()\ndef first(line):\n    return line.split()[0]\n\nif __name__ == '__main__':\n    cbox.main(first)\n```\n\nrunning it:\n\n```bash\n$ cat langs.txt | ./first.py \npython\nlisp\nruby\n```\n\n**or inline cli style:**\n\n```bash\n$ cat langs.txt | cbox 's.split()[0]'\n```\n\n*note: **`s`** is the input variable*\n\n\nnow dave is satisfied, so like every satisfied programmer - he wants more!\n\ndave now wants to get a list of the langs urls.\n\n### arguments and help message\n\n```python\n#!/usr/bin/env python3\n# nth-item.py\nimport cbox\n\n@cbox.stream()\n# we can pass default values and use type annotations for correct types\ndef nth_item(line, n: int = 0):\n    \"\"\"returns the nth item from each line.\n\n    :param n: the number of item position starting from 0\n    \"\"\"\n    return line.split()[n]\n\nif __name__ == '__main__':\n    cbox.main(nth_item)\n```\n\nrunning it:\n\n```bash\n#!/usr/bin/env python3\n$ ./nth-item.py --help\nusage: nth-item.py [-h] [-n N]\n\nreturns the nth item from each line.\n\noptional arguments:\n  -h, --help  show this help message and exit\n  -n N        the number of item position starting from 0\n```\n\n```bash\n$ cat langs.txt | ./nth-item.py \npython\nlisp\nruby\n```\n\n```bash\n$ cat langs.txt | ./nth-item.py -n 1\nhttp://python.org\nhttp://lisp-lang.org\nhttp://ruby-lang.org\n```\n\nnow dave wants to get the status out of each url, for this we can use `requests`.\n\nbut to process a large list it will take too long, so he better off use threads.\n\n### threading example\n\n```python\n#!/usr/bin/env python3\n# url-status.py\nimport cbox\nimport requests\n\n@cbox.stream(worker_type='thread', max_workers=4)\ndef url_status(line):\n    resp = requests.get(line)\n    return f'{line} - {resp.status_code}'\n\nif __name__ == '__main__':\n    cbox.main(url_status)\n```\n\n**running it:**\n\n```bash\n$ cat langs.txt | ./nth-line.py -n 1 | ./url-status.py \nhttp://python.org - 200\nhttp://lisp-lang.org - 200\nhttp://ruby-lang.org - 200\n```\n\n**or inline cli style**\n\n```bash\n$ cat langs.txt | cbox 's.split()[1]' | cbox -m requests  -w thread -c 4 'f\"{s} - {requests.get(s).status_code}\"'\nhttp://python.org - 200\nhttp://lisp-lang.org - 200\nhttp://ruby-lang.org - 200\n```\n\n\n## Advanced Usage\n### Error handling\n\n```python\n#!/usr/bin/env python3\n# numbersonly.py\nimport cbox\n\n@cbox.stream()\ndef numbersonly(line):\n    \"\"\"returns the lines containing only numbers. bad lines reported to stderr.\n    if any bad line is detected, exits with exitcode 2.\n    \"\"\"\n    if not line.isnumeric():\n        raise ValueError('{} is not a number'.format(line))\n    return line\n\nif __name__ == '__main__':\n    cbox.main(numbersonly)\n```\n\nall errors are redirected to `stderr`:\n\n```bash\n$ echo -e \"123\\nabc\\n567\" | ./numbersonly.py\n123\nTraceback (most recent call last):\n  File \"/home/shmulik/cs/cbox/cbox/concurrency.py\", line 54, in _simple_runner\n    yield func(item, **kwargs), None\n  File \"numbersonly.py\", line 11, in numbersonly\n    raise ValueError('{} is not a number'.format(line))\nValueError: abc is not a number\n\n567\n\n```\n\nwe can ignore the `stderr` stream by redirecting it to `/dev/null`:\n```bash\n$ echo -e \"123\\nabc\\n567\" | ./numbersonly.py 2\u003e/dev/null\n123\n567\n```\n\nour command returns 2 as the [exit code](https://en.wikipedia.org/wiki/Exit_status#Shell_and_scripts), \nindicating an error, we can get the last error code by running `echo $?`:\n\n```bash\n$ echo $?\n2\n```\n\n### Filtering\n\n`cbox.stream` supports three types of return values - `str`, `None` and `iterable` of `str`s.\n\n`None` skips and outputs nothing, `str` is outputted normally and each item in the `iterable` is treated as `str`.\n\nhere is a simple example:\n\n```python\n#!/usr/bin/env python3\n# extract-domains.py\nimport re\nimport cbox\n\n@cbox.stream()\ndef extract_domains(line):\n    \"\"\"tries to extract all the domains from the input using simple regex\"\"\"\n    return re.findall(r'(?:\\w+\\.)+\\w+', line) or None  # or None can be omitted\n\nif __name__ == '__main__':\n    cbox.main(extract_domains)\n```\n\nwe can now run it (notice that we can have multiple domains or zero domains on each line):\n```bash\n$ echo -e \"google.com cbox.com\\nhello\\nfacebook.com\" | ./extract-domains.py \ngoogle.com\ncbox.com\nfacebook.com\n```\n\n### Early Stopping\n`cbox.stream` supports early stopping, i.e. stopping before reading the whole `stdin`\n\nexample implementing a simple `head` command\n```python\n#!/usr/bin/env python3\n# head.py\nimport cbox\n\ncounter = 0\n\n\n@cbox.stream()\ndef head(line, n: int):\n    \"\"\"returns the first `n` lines\"\"\"\n    global counter\n    counter += 1\n\n    if counter \u003e n:\n        raise cbox.Stop()  # can also raise StopIteration()\n    return line\n\n\nif __name__ == '__main__':\n    cbox.main(head)\n```\n\ngetting the first 2 lines:\n\n```bash\n$ echo -e \"1\\n2\\n3\\n4\" | ./head.py -n 2\n1\n2\n```\n\n\n### Concurrency\n\n`cbox` supports **simple (default)**, **asyncio** and **thread** workers. we can use asyncio like this:\n\n```python\n#!/usr/bin/env python3\n# tcping.py\nimport asyncio\nimport cbox\n\n@cbox.stream(worker_type='asyncio', workers_window=30)\nasync def tcping(domain, timeout: int=3):\n    loop = asyncio.get_event_loop()\n\n    fut = asyncio.open_connection(domain, 80, loop=loop)\n    try:\n        reader, writer = await asyncio.wait_for(fut, timeout=timeout)\n        writer.close()\n        status = 'up'\n    except (OSError, asyncio.TimeoutError):\n        status = 'down'\n\n    return '{} is {}'.format(domain, status)\n\nif __name__ == '__main__':\n    cbox.main(tcping)\n```\n\nthis will try open up to 30 connections in parallel using asyncio. \n\nrunning it:\n\n```bash\n$ echo -e \"192.168.1.1\\n192.168.2.3\\ngoogle.com\"  | ./tcping.py\n192.168.1.1 is down\n192.168.2.3 is down\ngoogle.com is up\n```\n\n__more examples can be found on `examples/` dir__\n\n## Contributing\ncbox is an open source software and intended for everyone. please feel free to create PRs, add examples to examples/ dir, request features and ask questions.\n\n### Creating Local Dev Env\n\nafter cloning the repo, you'll need to install test dependencies from `test-requirements.txt`.\n\nthere is a simple `make` command to install them (you'll need [`miniconda`](https://conda.io/miniconda.html) installed):\n\n```bash\n$ make test-setup\n```\n\nor you can use `pip install -r test-requirements.txt` (preferably in new virtualenv).\n\nnow ensure all tests passes and runs locally:\n\n```bash\n$ make test\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshmuelamar%2Fcbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshmuelamar%2Fcbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshmuelamar%2Fcbox/lists"}