{"id":17798959,"url":"https://github.com/icy/dusybox","last_synced_at":"2025-03-17T05:32:02.329Z","repository":{"id":139397396,"uuid":"102318195","full_name":"icy/dusybox","owner":"icy","description":"I'm learning Dlang","archived":false,"fork":false,"pushed_at":"2023-11-14T07:24:18.000Z","size":108,"stargazers_count":12,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-27T21:22:12.071Z","etag":null,"topics":["dlang","learning","shell","system-programming"],"latest_commit_sha":null,"homepage":"","language":"D","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/icy.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-04T04:18:14.000Z","updated_at":"2023-11-05T19:30:45.000Z","dependencies_parsed_at":"2024-10-27T12:07:34.234Z","dependency_job_id":"2469b28f-598f-4e81-8b4c-6fd87c6ac663","html_url":"https://github.com/icy/dusybox","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/icy%2Fdusybox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/icy%2Fdusybox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/icy%2Fdusybox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/icy%2Fdusybox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/icy","download_url":"https://codeload.github.com/icy/dusybox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243852484,"owners_count":20358271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dlang","learning","shell","system-programming"],"created_at":"2024-10-27T12:03:42.597Z","updated_at":"2025-03-17T05:32:01.936Z","avatar_url":"https://github.com/icy.png","language":"D","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![tests](https://github.com/icy/dusybox/actions/workflows/ci.yaml/badge.svg)](https://github.com/icy/dusybox/actions/workflows/ci.yaml)\n\n## Description\n\nSimple implementations of system utilities in Dlang.\nThe primary purpose is to understand `Dlang`\nand to learn system programming.\nLearning notes are written in [NOTES.md](NOTES.md).\nList of tools:\n\n* [free](#free): display system memory by reading from `/proc`/.\n  Topics: Struct. Function overloading.\n* [watch](#watch): Watch command output, may exit if output matches some regular expression.\n  Topics: system command invocation, Getopt.\n  Links: [TODO][td1] and [Examples][ex1].\n* [plotbar](#plotbar): Draw 2-d bar chat, a tool inspired by https://github.com/lebinh/goplot.\n  Topics: Struct, Overloading, Testing.\n  Links: [TODO][td2] and [Examples][ex2].\n* [jq](#jq): Simple `json` reader, parses every json line from STDIN.\n  Topics: JSON parser.\n  Links: [TODO][td2] and [Examples][ex3].\n* [hello][Bhello]: Simple Hello builtin command writtedn in Dlang,\n  can be loaded into Bash environments.\n  Topics: Bash, C.\n* [json.validator][jv]: Match json input against a set of rules.\n  Topics: `json`, `opApply`, recursive.\n  For examples, please see the unittest within the source file.\n\n[jv]:  #json-validator\n[td1]: #todo-1\n[td2]: #todo-2\n[td3]: #todo-3\n[ex1]: #examples-1\n[ex2]: #examples-2\n[ex3]: #examples-3\n[Bhello]: #a-bash-builtin-command\n\n## Getting started\n\nTo use any utilities below, you need a `dlang` compiler and also\nthe `dub` package manager. The compiler can be installed by your\nsystem package manager, for e.g,\n\n```\n$ pacman -S dmd   # on ArchLinux\n```\n\nTo intsall `dub` please follow this link https://github.com/dlang/dub#installation.\nThe latest `ArchLinux` database has `dub` in the official repository and\nyou can also install them with `pacman -S dub`.\n\nNow from the top directory of project, you can start testing / running any tool as below\n\n```\n$ dub test -d 2 dusybox:free\n$ dub run dusybox:free\n```\n\nWhen being compiled, the tool's binary is located under `./output/` directory.\nFor example, `./output/dzfree` here `dz` is the common prefix for our tools\n(`dz` sounds a bit similar to `dusy`, doesn't it?).\nThe testing and other build profile still generate binary files\nunder the top directory though.\n\nThe `Makefile` can help to run all tests and/or to build release versions\nof all tools. You can override the list of tools with help of `TOOLS=`:\n\n```\n$ make releases TOOLS=watch\n```\n\n## json validator\n\nSource: [lib/dusybox/json/validator.d](lib/dusybox/json/validator.d).\n\nA simple json validator inspired by the Python code\nhttps://github.com/rycus86/webhook-proxy/blob/a8919cc82173b8e7a4cb0a2ba8a34a14996e159c/src/endpoints.py#L141.\nFor example, given a json data `input = {\"foo\": [0,1,\"2bar\"]}`\nand a rule, `{\"foo\": \"[0-9]+\"}`, the validator returns `true` as every\nitem in the value `input[\"foo\"]` matches against the regular expression\n`[0-9]+`. However, the validator returns `false` when the rule is updated\nto ``{\"foo\": \"[0-9]{2}\"}``\n\n## free\n\nPrint information about system memory.\n\nIt's similar to the `free` command on your `Linux` system.\n\n### TODO\n\n- [ ] Print Swap information _(used/total)_\n- [x] Support different Linux versions\n- [x] Print various information in a single command\n- [x] Print human-readable memory size\n\n### Examples\n\n```\n$ dub run dusybox:free\n                total        user        free      shared  buff/cache   available\nMem (kB):    16337688     3181032     6025164      759692     7131492    12487640\nMem (mB):       15954        3106        5883         741        6964       12194\nMem (gB):          15           3           5           0           6          11\nMem  (%):      100.00       19.47       36.88        4.65       43.65       76.43\n```\n\n## watch\n\nExecute a shell command and print its output to the standard output device\nevery one second. This is similar to the popular `watch` command.\n\n### TODO\n\n- [x] Document the external requirement (e.g, `libncursesw5-dev` on `Ubuntu-16.04`)\n- [x] Do not work inside `screen`. Work-around: `TERM=tmux dzwatch`.\n      See also https://github.com/D-Programming-Deimos/ncurses/issues/35.\n      To fix this issue, you need to rebuild the application on the\n      target machine.\n- [ ] Support `-i` to work with case-insensitive regular expression\n- [x] Redirect output from `stderr` (This works out-of-the-box)\n- [x] Print time information of the last interaction\n- [x] Print basic information about input command and iterator number.\n- [x] Wait 1 second after every execution. No more `--interval 1` :)\n- [x] Specify maxium number of executions with `-n \u003cnumber\u003e`\n- [x] Fix problem with overflow output, e.g, generated by `ps xauw`. Wait for https://github.com/mpevnev/nice-curses/issues/2.\n- [x] Exit if output matches some regular expression\n- [x] Exit if user presses `q` or `Q`. Wait for https://github.com/mpevnev/nice-curses/issues/1.\n- [x] Tool doesn't work with pipe commands, e.g, `ps x | grep foo`:\n      it reports `command not found` error. As a work-around you can\n      use `bash -c \"ps x | grep ff\"` instead.\n\n### Examples\n\n```\n$ dub run dusybox:watch -- free -m\n$ dub run dusybox:watch -- ps x\n\n$ dub run dusybox:watch -- -n 10 'ps xwa | grep \"f[i]r\"'\n:: No 4/10, Cmd [\"ps xwa | grep \\\"f[i]r\\\"\"]\n15774 ?        SNsl   3:01 /usr/lib/firefox/firefox\n15776 ?        ZN     0:00 [firefox] \u003cdefunct\u003e\n\n...\n:: Reached maximum number of interation (2) at 2017-Sep-05 18:04:47.0811478.\n```\n\nWatch `ElasticSearch` cluster status and exit if cluster is green:\n\n```\n$ dzwatch -e '\"status\"\\s+:\\s+\"green\"' 'curl -s http://elk.example.net:9201/_cluster/health | json_pp'\n```\n\n## plotbar\n\nThis tool is inspired by https://github.com/lebinh/goplot.\n\nIt visualizes your data as a simple [bar chart](https://en.wikipedia.org/wiki/Bar_chart).\n`goplot` draws relative bars _(compare bar height to the highest bar)_,\nwhy this tool draws absolute bars _(compare bar height to the sum of all bars)_.\n\nThe tool reads data from `STDIN` (the only source so far),\nand fetches every entry in format\n\n```\nkey value1\nkey with space   value2\n```\n\nIt will generate error messages to `STDERR` in case some line doesn't\nmatch the above format and/or their `value` is invalid.\n\n### TODO\n\n- [ ] Detect if there is any input data from `STDIN`\n- [ ] Option to draw relative bars\n- [ ] Continuous mode (keep drawing new bar while reading from `stdin`)\n- [x] Support tab delimeter in `key value` line\n- [ ] Support negative data (2-direction bar chart)\n- [x] Display actual value after the bar\n- [x] Set the minium percent number to display (`-m min`)\n- [ ] Display last n items (like `sort | tail`)\n- [ ] Sort the output (if the input is sorted)\n- [x] Additive mode (Sum of duplicated items)\n- [x] Fix bug when parsing input data (previous `value` is reused.)\n- [x] Move common part to a library file\n- [ ] Avoid overflow (when input key is too long, and/or the bar is too high)\n- [ ] Use `gnuplot` instead?\n- [x] Space in key name\n- [x] Support value on the first column\n- [ ] Exclude some key (with `-x` option)\n\n### Examples\n\nFind the biggest folder items, display ones consume great than `2%` of total storage.\n_(The idea for this example comes from https://github.com/lebinh/goplot.)_\nPlease note that you can't use `\\t` character in this example: The input parser\ndoesn't understand tab.\n\n```\n$ dub run dusybox:plotbar -- -m 2 \u003c \u003c(2\u003e/dev/null du -s /home/* | awk '{printf(\"%s %s\\n\", $2, $1)}')\n\n/home/pi.fast :  9 % ========= (9466072)\n     /home/pi : 13 % ============= (14541032)\n /home/btsync : 64 % ================================================================ (69425660)\n  /home/ebook :  8 % ======== (8600288)\n /home/backup :  2 % == (2615004)\n```\n\nDisplay the `ElasticSearch` indices the have most documents.\nSkip all indices that consumes less than `2%` in the total number of documents.\n\n```\n$ curl -s 'elk.example.net:9201/_cat/indices?h=index,docs.count' | ./output/dzplotbar -m 2\n\n           aws-lambda-test-uat-test-20170824 :  9 % ========= (4986415)\napi-gateway-execution-logs-test-uat-20170824 :  4 % ==== (2486179)\n           aws-lambda-test-uat-test-20170824 :  2 % == (1177304)\n           aws-lambda-test-dev-test-20170815 :  4 % ==== (2227446)\n```\n\nDisplay the biggest indexes (in stored size):\n\n```\n$ curl -s 'elk.example.net:9201/_cat/indices?h=index,store.size\u0026bytes=k' | ./output/dzplotbar -m 2\n\naws-lambda-test-uat-test-20170824 :  2 % == (2847921)\n                     emr-20170904 :  2 % == (3364511)\naws-lambda-test-uat-test-20170824 :  4 % ==== (5544297)\naws-lambda-test-uat-test-20170821 :  2 % == (2853427)\n```\n\nNow find the biggest source (by discarding date suffixes):\n\n```\n$ curl -s 'elk.example.net:9201/_cat/indices?h=index,store.size\u0026bytes=k' \\\n  | sed -re 's#-[0-9]{8}##g' \\\n  | ./output/dzplotbar -m 5 2\u003e/dev/null\n\n  aws-lambda-test-uat-test :  5 % ===== (3145751)\n                       emr : 11 % =========== (6974423)\n aws-lambda-test-uat-test2 : 11 % =========== (6622399)\ncloudtrail-defaultloggroup : 11 % =========== (6726637)\n```\n\nFind the package that has most files on `ArchLinux` system\n\n```\n$ pacman -Ql | grep -vE '/$' | awk '{printf(\"%s 1\\n\", $1 );}' | ./output/dzplotbar -m 2\n            evince :  2 % == (3058)\n           efl-git :  2 % == (3563)\n           python2 :  3 % === (4646)\nadwaita-icon-theme :  4 % ==== (5426)\n              mono :  2 % == (2443)\n             linux :  3 % === (3984)\n     linux-headers :  9 % ========= (12296)\n            python :  5 % ===== (6784)\n               ghc :  4 % ==== (5728)\n claws-mail-themes :  3 % === (4689)\n           openssl :  2 % == (3252)\n               qt4 :  3 % === (3825)\n              perl :  2 % == (2393)\n            libxcb :  2 % == (2371)\n           ncurses :  3 % === (3678)\n             cmake :  2 % == (2267)\n         man-pages :  2 % == (3491)\n               gcc :  2 % == (2198)\n```\n\nFind the biggest packages on `ArchLinux` system\n\n```\n$ pacman -Qi | awk '\n    /Name/ {printf(\"%s\", $NF);}\n    /Installed Size.+KiB/ {printf(\" %s\\n\", $(NF-1))}\n    /Installed Size.+MiB/ {printf(\" %s\\n\", $(NF-1) * 1024)}' \\\n  | ./dzplotbar  -m 2\n          mono :   4 % ==== (199782)\n    ghc-static :  17 % ================= (843428)\n       firefox :   3 % === (143370)\nlinux-firmware :   4 % ==== (206377)\n      chromium :   4 % ==== (212572)\n        python :   3 % === (131430)\n           ghc :   8 % ======== (425339)\n           gcc :   2 % == (119081)\n```\n\nGet simple statistics of your git repository\n\n```\ngit log --pretty=format:%aN | sort | uniq -c | sort -rn  | dzplotbar -r -m5\n        Ky-Anh Huynh :  37 % ===================================== (592)\n         some person :   9 % ========= (144)\n```\n\n## jq\n\nThis is not https://github.com/stedolan/jq.\n\nInstead, this tool reads line from `STDIN` and considers\neach line as a `JSON` string. This is useful as I need to process\nmultiple `JSON` lines from `nginx` and/or `ELK` system.\n\nIf input line can be parsed, the result will be printed to `stdout`\n_(if the tool has not any argument)_, or each item from arguments\nis looked up in the final `JSON` object. If the argument is\n\n```\n./output/dzjq .foo bar\n```\n\nthen the `.foo` is used as a lookup key, while `bar` is printed literally.\nIf the program fails to query a key `.foo`, it prints `foo` instead.\n\n### TODO\n\n- [ ] Detect if there is any input data from `STDIN`\n- [ ] Handle delimeter\n- [ ] Handle formatted string\n- [ ] Handle object other than integer and/or string\n- [ ] Nested key query\n- [ ] Advanced query with array support\n- [x] Move common methods to a library file\n- [ ] Option to flush `STDOUT` buffer on every processed input line\n- [x] Add unit tests\n- [x] Literraly support\n- [x] Process lines from `STDIN` as invidual documents.\n      See also https://github.com/stedolan/jq/issues/744.\n\n## Examples\n\nPrint key `.a` and `.b`, print `1` literally.\n\n```\n$ echo '{\"a\": 9, \"b\": {\"c\": 1}}' | dub run dusybox:jq -- .a 1 .b\n9       1       {\"c\":1}\n```\n\nPrint the original `JSON` string\n\n```\n$ echo '{\"a\": 9, \"b\": {\"c\": 1}}' | dub run dusybox:jq --\n'{\"a\": 9, \"b\": {\"c\": 1}}'\n```\n\nGenerate simple statistics from `nginx` access log file.\nThe format of log file is similar to\n  [this one](https://github.com/icy/docker/blob/fluentd/context/etc/nginx/nginx.conf).\n\n```\n$ dub run dusybox:jq -- .host 1 \u003c /home/pi/df/acces.log | ./output/dzplotbar -m 2\n     kibana.int.example.net : 25 % ========================= (269)\n    airflow.dev.example.net :  3 % === (33)\n    grafana.int.example.net : 70 % ====================================================================== (755)\nairflow.staging.example.net :  3 % === (28)\n```\n\nHow about the requests or statuses?\n\n```\n$ dub run dusybox:jq -- .request_uri 1 \u003c /home/pi/df/acces.log | ./output/dzplotbar -m 2\n/api/console/proxy?path=_aliases\u0026method=GET :  4 % ==== (44)\n/api/console/proxy?path=_mapping\u0026method=GET :  4 % ==== (44)\n                  /api/datasources/proxy/16 : 34 % ================================== (364)\n                  /api/datasources/proxy/14 : 12 % ============ (132)\n                  /api/datasources/proxy/13 :  5 % ===== (55)\n                    /elasticsearch/_msearch :  4 % ==== (40)\n                  /api/datasources/proxy/12 : 11 % =========== (122)\n\n$ dub run dusybox:jq -- .status 1 \u003c /home/pi/df/acces.log | ./output/dzplotbar -m 2\n200 : 93 % ============================================================================================= (1013)\n304 :  4 % ==== (43)\n```\n\n## A Bash builtin command\n\nWe can write `Bash` built-in command in `Dlang`.\nThanks a lot `evilrat` on `Dlang` forum for the idea.\n\n```\n$ dub build dusybox:bash_builtin_hello\n$ enable -f ./output/libdz_hello.so dz_hello\n\n$ type -a dz_hello\ndz_hello is a shell builtin\n\n$ dz_hello\nHello, world. It's Hello builtin command written in Dlang.\n\n$ help dz_hello\ndz_hello: dz_hello\n    Hello, it's from Dlang.\n\n    A Hello builtin command written in Dlang.\n\n$ enable -d dz_hello\n$ dz_hello\n-bash: dz_hello: command not found\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ficy%2Fdusybox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ficy%2Fdusybox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ficy%2Fdusybox/lists"}