Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/icy/dusybox
I'm learning Dlang
https://github.com/icy/dusybox
dlang learning shell system-programming
Last synced: 22 days ago
JSON representation
I'm learning Dlang
- Host: GitHub
- URL: https://github.com/icy/dusybox
- Owner: icy
- Created: 2017-09-04T04:18:14.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2023-11-14T07:24:18.000Z (about 1 year ago)
- Last Synced: 2024-04-22T00:22:22.987Z (7 months ago)
- Topics: dlang, learning, shell, system-programming
- Language: D
- Homepage:
- Size: 105 KB
- Stars: 12
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
[](https://github.com/icy/dusybox/actions/workflows/ci.yaml)
## Description
Simple implementations of system utilities in Dlang.
The primary purpose is to understand `Dlang`
and to learn system programming.
Learning notes are written in [NOTES.md](NOTES.md).
List of tools:
* [free](#free): display system memory by reading from `/proc`/.
Topics: Struct. Function overloading.
* [watch](#watch): Watch command output, may exit if output matches some regular expression.
Topics: system command invocation, Getopt.
Links: [TODO][td1] and [Examples][ex1].
* [plotbar](#plotbar): Draw 2-d bar chat, a tool inspired by https://github.com/lebinh/goplot.
Topics: Struct, Overloading, Testing.
Links: [TODO][td2] and [Examples][ex2].
* [jq](#jq): Simple `json` reader, parses every json line from STDIN.
Topics: JSON parser.
Links: [TODO][td2] and [Examples][ex3].
* [hello][Bhello]: Simple Hello builtin command writtedn in Dlang,
can be loaded into Bash environments.
Topics: Bash, C.
* [json.validator][jv]: Match json input against a set of rules.
Topics: `json`, `opApply`, recursive.
For examples, please see the unittest within the source file.
[jv]: #json-validator
[td1]: #todo-1
[td2]: #todo-2
[td3]: #todo-3
[ex1]: #examples-1
[ex2]: #examples-2
[ex3]: #examples-3
[Bhello]: #a-bash-builtin-command
## Getting started
To use any utilities below, you need a `dlang` compiler and also
the `dub` package manager. The compiler can be installed by your
system package manager, for e.g,
```
$ pacman -S dmd # on ArchLinux
```
To intsall `dub` please follow this link https://github.com/dlang/dub#installation.
The latest `ArchLinux` database has `dub` in the official repository and
you can also install them with `pacman -S dub`.
Now from the top directory of project, you can start testing / running any tool as below
```
$ dub test -d 2 dusybox:free
$ dub run dusybox:free
```
When being compiled, the tool's binary is located under `./output/` directory.
For example, `./output/dzfree` here `dz` is the common prefix for our tools
(`dz` sounds a bit similar to `dusy`, doesn't it?).
The testing and other build profile still generate binary files
under the top directory though.
The `Makefile` can help to run all tests and/or to build release versions
of all tools. You can override the list of tools with help of `TOOLS=`:
```
$ make releases TOOLS=watch
```
## json validator
Source: [lib/dusybox/json/validator.d](lib/dusybox/json/validator.d).
A simple json validator inspired by the Python code
https://github.com/rycus86/webhook-proxy/blob/a8919cc82173b8e7a4cb0a2ba8a34a14996e159c/src/endpoints.py#L141.
For example, given a json data `input = {"foo": [0,1,"2bar"]}`
and a rule, `{"foo": "[0-9]+"}`, the validator returns `true` as every
item in the value `input["foo"]` matches against the regular expression
`[0-9]+`. However, the validator returns `false` when the rule is updated
to ``{"foo": "[0-9]{2}"}``
## free
Print information about system memory.
It's similar to the `free` command on your `Linux` system.
### TODO
- [ ] Print Swap information _(used/total)_
- [x] Support different Linux versions
- [x] Print various information in a single command
- [x] Print human-readable memory size
### Examples
```
$ dub run dusybox:free
total user free shared buff/cache available
Mem (kB): 16337688 3181032 6025164 759692 7131492 12487640
Mem (mB): 15954 3106 5883 741 6964 12194
Mem (gB): 15 3 5 0 6 11
Mem (%): 100.00 19.47 36.88 4.65 43.65 76.43
```
## watch
Execute a shell command and print its output to the standard output device
every one second. This is similar to the popular `watch` command.
### TODO
- [x] Document the external requirement (e.g, `libncursesw5-dev` on `Ubuntu-16.04`)
- [x] Do not work inside `screen`. Work-around: `TERM=tmux dzwatch`.
See also https://github.com/D-Programming-Deimos/ncurses/issues/35.
To fix this issue, you need to rebuild the application on the
target machine.
- [ ] Support `-i` to work with case-insensitive regular expression
- [x] Redirect output from `stderr` (This works out-of-the-box)
- [x] Print time information of the last interaction
- [x] Print basic information about input command and iterator number.
- [x] Wait 1 second after every execution. No more `--interval 1` :)
- [x] Specify maxium number of executions with `-n `
- [x] Fix problem with overflow output, e.g, generated by `ps xauw`. Wait for https://github.com/mpevnev/nice-curses/issues/2.
- [x] Exit if output matches some regular expression
- [x] Exit if user presses `q` or `Q`. Wait for https://github.com/mpevnev/nice-curses/issues/1.
- [x] Tool doesn't work with pipe commands, e.g, `ps x | grep foo`:
it reports `command not found` error. As a work-around you can
use `bash -c "ps x | grep ff"` instead.
### Examples
```
$ dub run dusybox:watch -- free -m
$ dub run dusybox:watch -- ps x
$ dub run dusybox:watch -- -n 10 'ps xwa | grep "f[i]r"'
:: No 4/10, Cmd ["ps xwa | grep \"f[i]r\""]
15774 ? SNsl 3:01 /usr/lib/firefox/firefox
15776 ? ZN 0:00 [firefox]
...
:: Reached maximum number of interation (2) at 2017-Sep-05 18:04:47.0811478.
```
Watch `ElasticSearch` cluster status and exit if cluster is green:
```
$ dzwatch -e '"status"\s+:\s+"green"' 'curl -s http://elk.example.net:9201/_cluster/health | json_pp'
```
## plotbar
This tool is inspired by https://github.com/lebinh/goplot.
It visualizes your data as a simple [bar chart](https://en.wikipedia.org/wiki/Bar_chart).
`goplot` draws relative bars _(compare bar height to the highest bar)_,
why this tool draws absolute bars _(compare bar height to the sum of all bars)_.
The tool reads data from `STDIN` (the only source so far),
and fetches every entry in format
```
key value1
key with space value2
```
It will generate error messages to `STDERR` in case some line doesn't
match the above format and/or their `value` is invalid.
### TODO
- [ ] Detect if there is any input data from `STDIN`
- [ ] Option to draw relative bars
- [ ] Continuous mode (keep drawing new bar while reading from `stdin`)
- [x] Support tab delimeter in `key value` line
- [ ] Support negative data (2-direction bar chart)
- [x] Display actual value after the bar
- [x] Set the minium percent number to display (`-m min`)
- [ ] Display last n items (like `sort | tail`)
- [ ] Sort the output (if the input is sorted)
- [x] Additive mode (Sum of duplicated items)
- [x] Fix bug when parsing input data (previous `value` is reused.)
- [x] Move common part to a library file
- [ ] Avoid overflow (when input key is too long, and/or the bar is too high)
- [ ] Use `gnuplot` instead?
- [x] Space in key name
- [x] Support value on the first column
- [ ] Exclude some key (with `-x` option)
### Examples
Find the biggest folder items, display ones consume great than `2%` of total storage.
_(The idea for this example comes from https://github.com/lebinh/goplot.)_
Please note that you can't use `\t` character in this example: The input parser
doesn't understand tab.
```
$ dub run dusybox:plotbar -- -m 2 < <(2>/dev/null du -s /home/* | awk '{printf("%s %s\n", $2, $1)}')
/home/pi.fast : 9 % ========= (9466072)
/home/pi : 13 % ============= (14541032)
/home/btsync : 64 % ================================================================ (69425660)
/home/ebook : 8 % ======== (8600288)
/home/backup : 2 % == (2615004)
```
Display the `ElasticSearch` indices the have most documents.
Skip all indices that consumes less than `2%` in the total number of documents.
```
$ curl -s 'elk.example.net:9201/_cat/indices?h=index,docs.count' | ./output/dzplotbar -m 2
aws-lambda-test-uat-test-20170824 : 9 % ========= (4986415)
api-gateway-execution-logs-test-uat-20170824 : 4 % ==== (2486179)
aws-lambda-test-uat-test-20170824 : 2 % == (1177304)
aws-lambda-test-dev-test-20170815 : 4 % ==== (2227446)
```
Display the biggest indexes (in stored size):
```
$ curl -s 'elk.example.net:9201/_cat/indices?h=index,store.size&bytes=k' | ./output/dzplotbar -m 2
aws-lambda-test-uat-test-20170824 : 2 % == (2847921)
emr-20170904 : 2 % == (3364511)
aws-lambda-test-uat-test-20170824 : 4 % ==== (5544297)
aws-lambda-test-uat-test-20170821 : 2 % == (2853427)
```
Now find the biggest source (by discarding date suffixes):
```
$ curl -s 'elk.example.net:9201/_cat/indices?h=index,store.size&bytes=k' \
| sed -re 's#-[0-9]{8}##g' \
| ./output/dzplotbar -m 5 2>/dev/null
aws-lambda-test-uat-test : 5 % ===== (3145751)
emr : 11 % =========== (6974423)
aws-lambda-test-uat-test2 : 11 % =========== (6622399)
cloudtrail-defaultloggroup : 11 % =========== (6726637)
```
Find the package that has most files on `ArchLinux` system
```
$ pacman -Ql | grep -vE '/$' | awk '{printf("%s 1\n", $1 );}' | ./output/dzplotbar -m 2
evince : 2 % == (3058)
efl-git : 2 % == (3563)
python2 : 3 % === (4646)
adwaita-icon-theme : 4 % ==== (5426)
mono : 2 % == (2443)
linux : 3 % === (3984)
linux-headers : 9 % ========= (12296)
python : 5 % ===== (6784)
ghc : 4 % ==== (5728)
claws-mail-themes : 3 % === (4689)
openssl : 2 % == (3252)
qt4 : 3 % === (3825)
perl : 2 % == (2393)
libxcb : 2 % == (2371)
ncurses : 3 % === (3678)
cmake : 2 % == (2267)
man-pages : 2 % == (3491)
gcc : 2 % == (2198)
```
Find the biggest packages on `ArchLinux` system
```
$ pacman -Qi | awk '
/Name/ {printf("%s", $NF);}
/Installed Size.+KiB/ {printf(" %s\n", $(NF-1))}
/Installed Size.+MiB/ {printf(" %s\n", $(NF-1) * 1024)}' \
| ./dzplotbar -m 2
mono : 4 % ==== (199782)
ghc-static : 17 % ================= (843428)
firefox : 3 % === (143370)
linux-firmware : 4 % ==== (206377)
chromium : 4 % ==== (212572)
python : 3 % === (131430)
ghc : 8 % ======== (425339)
gcc : 2 % == (119081)
```
Get simple statistics of your git repository
```
git log --pretty=format:%aN | sort | uniq -c | sort -rn | dzplotbar -r -m5
Ky-Anh Huynh : 37 % ===================================== (592)
some person : 9 % ========= (144)
```
## jq
This is not https://github.com/stedolan/jq.
Instead, this tool reads line from `STDIN` and considers
each line as a `JSON` string. This is useful as I need to process
multiple `JSON` lines from `nginx` and/or `ELK` system.
If input line can be parsed, the result will be printed to `stdout`
_(if the tool has not any argument)_, or each item from arguments
is looked up in the final `JSON` object. If the argument is
```
./output/dzjq .foo bar
```
then the `.foo` is used as a lookup key, while `bar` is printed literally.
If the program fails to query a key `.foo`, it prints `foo` instead.
### TODO
- [ ] Detect if there is any input data from `STDIN`
- [ ] Handle delimeter
- [ ] Handle formatted string
- [ ] Handle object other than integer and/or string
- [ ] Nested key query
- [ ] Advanced query with array support
- [x] Move common methods to a library file
- [ ] Option to flush `STDOUT` buffer on every processed input line
- [x] Add unit tests
- [x] Literraly support
- [x] Process lines from `STDIN` as invidual documents.
See also https://github.com/stedolan/jq/issues/744.
## Examples
Print key `.a` and `.b`, print `1` literally.
```
$ echo '{"a": 9, "b": {"c": 1}}' | dub run dusybox:jq -- .a 1 .b
9 1 {"c":1}
```
Print the original `JSON` string
```
$ echo '{"a": 9, "b": {"c": 1}}' | dub run dusybox:jq --
'{"a": 9, "b": {"c": 1}}'
```
Generate simple statistics from `nginx` access log file.
The format of log file is similar to
[this one](https://github.com/icy/docker/blob/fluentd/context/etc/nginx/nginx.conf).
```
$ dub run dusybox:jq -- .host 1 < /home/pi/df/acces.log | ./output/dzplotbar -m 2
kibana.int.example.net : 25 % ========================= (269)
airflow.dev.example.net : 3 % === (33)
grafana.int.example.net : 70 % ====================================================================== (755)
airflow.staging.example.net : 3 % === (28)
```
How about the requests or statuses?
```
$ dub run dusybox:jq -- .request_uri 1 < /home/pi/df/acces.log | ./output/dzplotbar -m 2
/api/console/proxy?path=_aliases&method=GET : 4 % ==== (44)
/api/console/proxy?path=_mapping&method=GET : 4 % ==== (44)
/api/datasources/proxy/16 : 34 % ================================== (364)
/api/datasources/proxy/14 : 12 % ============ (132)
/api/datasources/proxy/13 : 5 % ===== (55)
/elasticsearch/_msearch : 4 % ==== (40)
/api/datasources/proxy/12 : 11 % =========== (122)
$ dub run dusybox:jq -- .status 1 < /home/pi/df/acces.log | ./output/dzplotbar -m 2
200 : 93 % ============================================================================================= (1013)
304 : 4 % ==== (43)
```
## A Bash builtin command
We can write `Bash` built-in command in `Dlang`.
Thanks a lot `evilrat` on `Dlang` forum for the idea.
```
$ dub build dusybox:bash_builtin_hello
$ enable -f ./output/libdz_hello.so dz_hello
$ type -a dz_hello
dz_hello is a shell builtin
$ dz_hello
Hello, world. It's Hello builtin command written in Dlang.
$ help dz_hello
dz_hello: dz_hello
Hello, it's from Dlang.
A Hello builtin command written in Dlang.
$ enable -d dz_hello
$ dz_hello
-bash: dz_hello: command not found
```