https://github.com/mk-fg/fgtk

A set of a misc tools to work with files and processes
https://github.com/mk-fg/fgtk

bash collection dev distro git linux metrics misc python scraps sysadmin systemd tools utilities

Last synced: about 1 year ago
JSON representation

A set of a misc tools to work with files and processes

Host: GitHub
URL: https://github.com/mk-fg/fgtk
Owner: mk-fg
Created: 2012-03-26T09:58:03.000Z (about 14 years ago)
Default Branch: master
Last Pushed: 2024-10-29T19:03:52.000Z (over 1 year ago)
Last Synced: 2024-10-29T19:04:19.362Z (over 1 year ago)
Topics: bash, collection, dev, distro, git, linux, metrics, misc, python, scraps, sysadmin, systemd, tools, utilities
Language: Python
Homepage:
Size: 3.45 MB
Stars: 157
Watchers: 12
Forks: 34
Open Issues: 0
Metadata Files:
- Readme: README.md
- Audit: audit-follow

Awesome Lists containing this project

README

# fgtk

A set of a misc tools to work with files and processes.

Various oldish helper scripts/binaries I wrote to help myself with
day-to-day tasks.

License for all scripts is WTFPL (public domain-ish - [see below](#hdr-license__wtfpl_)),
feel free to just copy and use these in whatever way you like.

Repository URLs:

-
-
-

Contents - links to doc section for each script here:

- [\[-root-\] Various CLI/system things](#hdr--root-___various_cli_system_things)

- [File/dir/fs management](#hdr-file_dir_fs_management)

- [scim](#hdr-scim)
- [run_cmd_pipe.nim](#hdr-run_cmd_pipe.nim)
- [findx](#hdr-findx)
- [patch-nspawn-ids](#hdr-patch-nspawn-ids)
- [bindfs-idmap](#hdr-bindfs-idmap)
- [docker-ln](#hdr-docker-ln)
- [fast-disk-wipe](#hdr-fast-disk-wipe)
- [lsx](#hdr-lsx)
- [trunc-filenames](#hdr-trunc-filenames)

- [Various file-data processing tools](#hdr-various_file-data_processing_tools)

- [repr](#hdr-repr)
- [color](#hdr-color)
- [resolve-hostnames](#hdr-resolve-hostnames)
- [resolve-conf](#hdr-resolve-conf)
- [temp-patch](#hdr-temp-patch)
- [term-pipe](#hdr-term-pipe)
- [yaml-to-pretty-json](#hdr-yaml-to-pretty-json)
- [yaml-flatten](#hdr-yaml-flatten)
- [yaml-diff](#hdr-yaml-diff)
- [hz](#hdr-hz)
- [liac](#hdr-liac)
- [html-embed](#hdr-html-embed)
- [someml-indent](#hdr-someml-indent)
- [hashname](#hdr-hashname)
- [hhash](#hdr-hhash)
- [crypt](#hdr-crypt)

- [Kernel sources/build/version management](#hdr-kernel_sources_build_version_management)

- [kernel-patch](#hdr-kernel-patch)
- [kernel-conf-check](#hdr-kernel-conf-check)
- [clean-boot](#hdr-clean-boot)

- [ZNC log helpers](#hdr-znc_log_helpers)

- [znc-log-aggregator](#hdr-znc-log-aggregator)
- [znc-log-reader](#hdr-znc-log-reader)

- [systemd-related](#hdr-systemd-related)

- [systemd-dashboard](#hdr-systemd-dashboard)
- [systemd-watchdog](#hdr-systemd-watchdog)
- [cgrc](#hdr-cgrc)

- [SSH and WireGuard related](#hdr-ssh_and_wireguard_related)

- [ssh-fingerprint](#hdr-ssh-fingerprint)
- [ssh-keyparse](#hdr-ssh-keyparse)
- [ssh-key-init](#hdr-ssh-key-init)
- [ssh-tunnel](#hdr-ssh-tunnel)
- [ssh-reverse-mux-server / ssh-reverse-mux-client](#hdr-ssh-reverse-mux-server_ssh-reverse-mux-client)
- [wg-mux-server / wg-mux-client](#hdr-wg-mux-server___wg-mux-client)
- [ssh-tunnels-cleanup](#hdr-ssh-tunnels-cleanup)
- [mosh-nat / mosh-nat-bind.c](#hdr-mosh-nat___mosh-nat-bind.c)
- [tping](#hdr-tping)

- [WiFi / Bluetooth helpers](#hdr-wifi___bluetooth_helpers)

- [adhocapd](#hdr-adhocapd)
- [wpa-systemd-wrapper](#hdr-wpa-systemd-wrapper)
- [timed-ble-beacon](#hdr-timed-ble-beacon)
- [timed-ble-beacon-mpy-led](#hdr-timed-ble-beacon-mpy-led)

- [Misc](#hdr-misc)

- [at](#hdr-at)
- [sleepc](#hdr-sleepc)
- [wgets](#hdr-wgets)
- [mail](#hdr-mail)
- [passgen](#hdr-passgen)
- [urlparse](#hdr-urlparse)
- [ip-ext](#hdr-ip-ext)
- [blinky](#hdr-blinky)
- [openssl-fingerprint](#hdr-openssl-fingerprint)
- [nsh](#hdr-nsh)
- [pam-run](#hdr-pam-run)
- [primes](#hdr-primes)
- [boot-patcher](#hdr-boot-patcher)
- [audit-follow](#hdr-audit-follow)
- [tui-binary-conv](#hdr-tui-binary-conv)
- [maildir-cat](#hdr-maildir-cat)
- [dns-update-proxy](#hdr-dns-update-proxy)
- [dns-test-daemon](#hdr-dns-test-daemon)
- [nginx-access-log-stat-block](#hdr-nginx-access-log-stat-block)
- [sys-wait](#hdr-sys-wait)
- [yt-feed-to-email](#hdr-yt-feed-to-email)
- [color-b64sort](#hdr-color-b64sort)
- [svg-tweak](#hdr-svg-tweak)
- [unix-socket-links](#hdr-unix-socket-links)
- [tcpdump-translate](#hdr-tcpdump-translate)

- [\[dev\] Dev tools](#hdr-dev___dev_tools)

- [indent-replace](#hdr-indent-replace)
- [indent-braces](#hdr-indent-braces)
- [golang_filter](#hdr-golang_filter)
- [distribute_regen](#hdr-distribute_regen)
- [darcs_bundle_to_diff](#hdr-darcs_bundle_to_diff)
- [git-nym](#hdr-git-nym)
- [git-meld](#hdr-git-meld)
- [catn](#hdr-catn)
- [git_terminate](#hdr-git_terminate)
- [git_contains](#hdr-git_contains)
- [gtk-val-slider](#hdr-gtk-val-slider)
- [git-version-bump-filter](#hdr-git-version-bump-filter)
- [git-prepare-commit-msg-hook](#hdr-git-prepare-commit-msg-hook)
- [markdown-checks](#hdr-markdown-checks)

- [\[backup\] Backup helpers](#hdr-backup___backup_helpers)

- [ssh-r-sync / ssh-r-sync-recv](#hdr-ssh-r-sync___ssh-r-sync-recv)
- [ssh-dump](#hdr-ssh-dump)
- [zfs-snapper](#hdr-zfs-snapper)
- [btrfs-snapper](#hdr-btrfs-snapper)
- [dir-snapper](#hdr-dir-snapper)

- [\[hsm\] FIDO2 / PIV / etc smartcard stuff](#hdr-hsm___fido2___piv___etc_smartcard_stuff)

- [fido2-hmac-desalinate.c](#hdr-fido2-hmac-desalinate.c)
- [fido2_hmac_boot.nim](#hdr-fido2_hmac_boot.nim)
- [secret-token-backup](#hdr-secret-token-backup)

- [\[desktop\] Linux desktop stuff](#hdr-desktop___linux_desktop_stuff)

- [\[desktop/uri_handlers\]](#hdr-desktop_uri_handlers__)

- [\[desktop/media\]](#hdr-desktop_media__)

- [toogg](#hdr-toogg)
- [tomkv](#hdr-tomkv)
- [totty](#hdr-totty)
- [split](#hdr-split)
- [audio-split-m4b](#hdr-audio-split-m4b)
- [audio-split-flac-cue](#hdr-audio-split-flac-cue)
- [video-concat-xfade](#hdr-video-concat-xfade)
- [pick-tracks](#hdr-pick-tracks)
- [twitch_vod_fetch](#hdr-twitch_vod_fetch)
- [ytdl-chan](#hdr-ytdl-chan)
- [streamdump](#hdr-streamdump)
- [image-compact](#hdr-image-compact)

- [\[desktop/notifications\]](#hdr-desktop_notifications__)

- [exec](#hdr-exec)
- [power](#hdr-power)
- [logtail](#hdr-logtail)
- [dovecot-mail](#hdr-dovecot-mail)
- [icon](#hdr-icon)
- [aqm-alerts](#hdr-aqm-alerts)
- [dev-nodes](#hdr-dev-nodes)

- [\[desktop\] others](#hdr-desktop___others)

- [vfat_shuffler](#hdr-vfat_shuffler)
- [fan_control](#hdr-fan_control)
- [emms-beets-enqueue](#hdr-emms-beets-enqueue)
- [ff_backup](#hdr-ff_backup)
- [ff-cli](#hdr-ff-cli)
- [bt_agent](#hdr-bt_agent)
- [alarm](#hdr-alarm)
- [acpi-wakeup-config](#hdr-acpi-wakeup-config)
- [olaat](#hdr-olaat)
- [blinds](#hdr-blinds)
- [evdev-to-xev](#hdr-evdev-to-xev)
- [exclip](#hdr-exclip)
- [xdpms](#hdr-xdpms)
- [xiwait](#hdr-xiwait)
- [xkbledq](#hdr-xkbledq)
- [rss-get](#hdr-rss-get)
- [qr](#hdr-qr)
- [gtk-color-calc](#hdr-gtk-color-calc)
- [filetag](#hdr-filetag)
- [hamster-tally](#hdr-hamster-tally)
- [feh-screen](#hdr-feh-screen)

- [\[vm\] VM scripts](#hdr-vm___vm_scripts)

- [\[bpf\] Linux eBPF filters](#hdr-bpf___linux_ebpf_filters)

- [\[arch\] ArchLinux(+ARM)](#hdr-arch___archlinux__arm_)

- [elf-deps](#hdr-elf-deps)
- [pacman-manifest](#hdr-pacman-manifest)
- [pacman-extra-files](#hdr-pacman-extra-files)
- [pacman-pacnew](#hdr-pacman-pacnew)
- [pacman-fsck](#hdr-pacman-fsck)
- [tar-strap](#hdr-tar-strap)
- [can-strap](#hdr-can-strap)

- [\[alpine\] Alpine Linux](#hdr-alpine___alpine_linux)

- [manifest](#hdr-manifest)

- [\[metrics\] Charts and metrics](#hdr-metrics___charts_and_metrics)

- [rrd-sensors-logger](#hdr-rrd-sensors-logger)
- [graphite-scratchpad](#hdr-graphite-scratchpad)
- [gnuplot-free](#hdr-gnuplot-free)
- [d3-line-chart-boilerplate](#hdr-d3-line-chart-boilerplate)
- [d3-histogram-boilerplate](#hdr-d3-histogram-boilerplate)
- [d3-temp-rh-sensor-tsv-series-chart](#hdr-d3-temp-rh-sensor-tsv-series-chart)
- [d3-du-disk-space-usage-layout](#hdr-d3-du-disk-space-usage-layout)
- [prometheus-snmp-iface-counters-exporter](#hdr-prometheus-snmp-iface-counters-exporter)
- [prometheus-grafana-simplejson-aggregator](#hdr-prometheus-grafana-simplejson-aggregator)
- [systemd-cglog](#hdr-systemd-cglog)
- [load-check-logger](#hdr-load-check-logger)

- [\[cron-checks\] Diff/alert checks for crontab](#hdr-cron-checks_things_to_run_from_crontab_s.0UQD)

- [df](#hdr-df)
- [attrs](#hdr-attrs)
- [git-manifest](#hdr-git-manifest)
- [systemd](#hdr-systemd)

- [\[scraps\]](#hdr-scraps__)

- [rsync-diff](#hdr-rsync-diff)
- [pcap-process](#hdr-pcap-process)
- [log-tail-check](#hdr-log-tail-check)
- [resize-rpi-fat32-for-card](#hdr-resize-rpi-fat32-for-card)
- [asciitree-parse](#hdr-asciitree-parse)
- [glusterfs-xattr-trusted-to-user](#hdr-glusterfs-xattr-trusted-to-user)
- [led-blink-arg](#hdr-led-blink-arg)
- [led-blink-seq](#hdr-led-blink-seq)
- [gue-tunnel](#hdr-gue-tunnel)
- [wifi-client-match](#hdr-wifi-client-match)
- [mem-search-replace](#hdr-mem-search-replace)
- [gpm-track](#hdr-gpm-track)
- [rsyslogs](#hdr-rsyslogs)
- [relp-test](#hdr-relp-test)
- [exec.c](#hdr-exec.c)
- [sqlite-python-concurrency-test](#hdr-sqlite-python-concurrency-test)
- [numfmt.awk](#hdr-numfmt.awk)
- [nft-ddos](#hdr-nft-ddos)

## Scripts

### \[-root-\] Various CLI/system things

#### File/dir/fs management

File/link/dir and filesystem structure manipulation tools.

##### [scim](scim)

Non-interactive CLI tool to keep a list of files to symlink or copy into/from
some "dotfiles" configuration dir or repository, and keep/check/update/restore
metadata manifest for these files.

Keeps track of ACLs, POSIX capabilities and xattrs for metadata, runs file
diffs for file copies and links, supports a bunch of neat symlinking options
(like using relative symlinks, relative symlinks into symlinked repo-dir, etc).

Idea is to keep links and metadata manifest files in some configuration repo,
and run the tool occasionally after system updates or manual changes to pull
updated files into repo, update files on fs from the repo, fix links/permissions
on fs, copy/add new ones, etc - all manifest/maintenance ops done via this script.

Format for links-list looks something like this:

.gitconfig -> .git/config
/usr/share/zoneinfo/Asia/Yekaterinburg -> /etc/localtime
bpf -> /etc/bpf
zshrc > /etc/zsh/zshrc
kernel-config > /usr/src/linux/.config
myapp/secret.conf -> /etc/myapp/secret.conf
myapp/suid.bin -> /usr/local/bin/myapp
myapp/caps.bin -> /usr/local/bin/myapp-helper

And metadata is also a simple plaintext file, with fancier stuff towards the
end of lines, on paths where it's used/needed:

.gitconfig root:root:644
bpf root:wheel:750
zshrc root:root:644
kernel-config root:wheel:664
myapp/secret.conf root:root:600
myapp/suid.bin root:root:4711
myapp/caps.bin root:root:4700/EP:net_raw/u::rwx,u:netuser:--x,g::r-x,m::r-x,o::---

In addition to lists, there're separate links/meta exclude-files with regexps of
paths to not warn about being missing in links-list or track metadata for.

Only needs python3 to run, has bundled implementation for parsing/encoding
modern linux ACLs/capabilities extended attributes.
Uses `git diff --no-index` for `--diff-cmd` by default, as it is very fast,
has nice colors and should be widely available.

Started as a [cfgit project] long time ago, evolved away into this more generic
(and not necessarily git-related) tool.

[cfgit project]: https://fraggod.net/code/git/configit/

##### [run_cmd_pipe.nim](run_cmd_pipe.nim)

Small tool to match lines from stdin according to ini config file
and run commands for any matching regexps specified there.
Intended as a long-running handler for monitoring some process' output,
e.g. monitor some log via `tail -F file.log`, or react to [fanotify]
filesystem updates from [fatrace] efficiently.

For example, with `myapp-changes.conf` file like this:

``` ini
# Add 10s delay for changes to settle before running commands
delay = 10_000

[data-file-updates]
regexp = : \S*[WD+<>]\S* */srv/myapp/data-files(/[^/]+)?$
run = myapp process-new-data /srv/myapp/data-files
# regexp-env-var = RCP_MATCH -- "run" command will get this in env by default
# regexp-env-group = 1 -- regexp group to put into regexp-env-var, 0 - full match
# regexp-run-group = 1 -- to run/delay/cooldown commands based on matched group

[config-updates]
regexp = : \S*[WD+<>]\S* */srv/myapp/config(/.*)?$
run = pkill -x HUP myapp
```

...tool can be run as `fatrace | run_cmd_pipe myapp-changes.conf` (or exec
input-command without shell via `... -- cmd args...` by itself), to process
any file-change events and run relevant commands to react to those in a daemon loop.

Can have cooldown and debouncing delay for rules, reloads config-file on SIGHUP,
runs only one process per rule at a time, has small mem footprint, no deps, etc etc.
`-h/--help` output has more info on configuration format and cli opts.

Build with:
`nim c -d:release --opt:size run_cmd_pipe.nim && strip run_cmd_pipe`

One interesting use I've found in combination with [fatrace] is to
[monitor and synchronize local containers, as well as handle events from those].

[fatrace]: https://github.com/martinpitt/fatrace
[fanotify]: https://lwn.net/Articles/339253/
[monitor and synchronize local containers, as well as handle events from those]:
https://blog.fraggod.net/2024/01/09/ab-using-fanotify-as-a-container-eventmessage-bus.html

##### [findx](findx)

Wrapper around GNU find (from [findutils]) to accept paths at the end of argv
if none are passed before query.

Makes it somewhat more consistent with most other commands that accept
options and a lists of paths (almost always after opts),
but still warns when/if reordering takes place.

No matter how many years I'm using that tool, still sometimes type paths
after query there, so decided to patch around that frustrating issue one day.

[findutils]: https://www.gnu.org/software/findutils/

##### [patch-nspawn-ids](patch-nspawn-ids)

Python script to "shift" or "patch" uid/gid values with new container-id
according to systemd-nspawn schema, i.e. set upper 16-bit to specified
container-id value and keep lower 16 bits to uid/gid inside the container.

Similar operation to what systemd-nspawn's --private-users-chown option does
(described in nspawn-patch-uid.c), but standalone, doesn't bother with ACLs or
checks on filesystem boundaries.

Main purpose is to update uids when migrating systemd-nspawn containers or
adding paths/filesystems to these without clobbering ownership info there.

Should be safe to use anywhere, as in most non-nspawn cases upper bits of
uid/gid are always zero, hence any changes can be easily reverted by running
this tool again with -c0.

##### [bindfs-idmap](bindfs-idmap)

[bindfs] wrapper script to setup id-mapping from uid of the mountpoint
to uid/gid of the source directory.

I.e. after `bindfs-idmap /var/lib/machines/home/src-user ~dst-user/tmp`,
`~dst-user/tmp` will be accessible to dst-user as if they were src-user,
with all operations proxied to src-user's dir.

Anything created under `~dst-user/tmp` will have uid/gid of the src dir.

Useful to allow temporary access to some uid's files in a local container
to a user id in a main namespace.

For long-term access (e.g. for some daemon), there probably are better options
than such bindfs hack - e.g. bind/idmapped mounts, shared uids/gids, ACLs, etc.

[bindfs]: https://bindfs.org/

##### [docker-ln](docker-ln)

Simple bash script to symlink uppermost "merged" overlayfs layer of a running
docker-compose setup container, to allow easy access to temporary files there.

Useful for testing stuff without the need to rebuild and restart whole container
or a bunch of compose stuff after every one-liner tweak to some script that's
supposed to be running in there, or to experiment-with and debug things.

These paths are very likely to change between container and docker-compose
restarts for many reasons, so such symlinks are generally only valid during
container runtime, and script needs a re-run to update these too.

##### [fast-disk-wipe](fast-disk-wipe.c)

Very simple "write 512B, skip N * 512B, repeat" binary for wiping some block
device in a hurry.

Idea is not to erase every trace of data or to hide it, but just to make files
probabilistically unusable due to such junk blocks all over the place.

With low-enough intervals it should also corrupt filesystem pretty badly,
making metadata hard to access.

Fast loop of 512B writes to a device directly will likely hang that binary until
it's done, as that's how such direct I/O seem to work on linux.

Writes only stop when write() or lseek() starts returning errors, so using this
on some extendable file will result in it eating up all space available to it.

See head of the file for build and usage info.

##### [lsx](lsx)

More functionality similar to common "ls" tool, to list files in some specific
ways that are occasionally useful. All those are available via various options -
see `-h/--help` for a full list.

For example, to print up to N `-a/--adjacent` files (within some specific ordering):
``` console
% lsx -aS data/chunk-12345.bin # default up to 10 before/after, w/ S=size ordering
% lsx -a 50as data/chunk-13.bin # only 50 files larger than specified one
% lsx -a 5bt myapp/state.log # up to 5 logs right before state.log by mtime
% lsx -fa a3 logs/20230515.log # 3 log-files (-f/--files) with names after that one
```

Or files within `-t/--mtime` vicinity/ranges:
``` console
% lsx -t 1h cache/a/bcdefg.json # files created/changed within 1h of that one
% lsx -t 5d/10d cache/*/* # mtime in 5d-10d ago range
% lsx -rt 2024-10-20/2024-10-25 # between those dates in the current dir
% lsx -rt 1am/3:30 logs # logs changed from 1am to 3:30am earlier today
```

Simple python script with no extra dependencies.

##### [trunc-filenames](trunc-filenames)

Python script to recursively shorten (truncate) file/directory names
under specified byte-limit, respecting typical filename format, suffixes
and multibyte encodings.

Useful for transferring files from NTFS and similar filesystems
to POSIX/linux ones that have strict 255-byte filename-length limit,
where non-english paths can get very long fast bytewise.

Truncates names decoded to unicode characters to avoid splitting those,
has somewhat complicated rules for how to truncate filenames with dot-suffixes
and multiple dots in them, disambiguates rename destinations on conflicts,
always keeps longest filename possible under `-l/--max-len` limit,
inserts unicode-ellipsis (…) character to indicate where truncation was made.

Defaults to dry-run mode for safety, only printing all renames to be made.

#### Various file-data processing tools

Things that manipulate some kind of data formats or mangle generic file/pipe contents.

##### [repr](repr)

Ever needed to check if file has newlines or BOM in it, yet every editor is
user-friendly by default and hides these from actual file contents?

One fix is hexdump or switching to binary mode, but these are usually terrible
for looking at text, and tend to display all non-ASCII as "." instead of nicer
\\r \\t \\n ... escapes, not to mention unicode chars.

This trivial script prints each line in a file via python's repr(), which is
usually very nice, has none of the above issues and doesn't dump byte codes on
you for anything it can interpret as char/codepoint or some neat escape code.

Has opts for text/byte mode and stripping "universal newlines" (see newline= in
built-in open() func).

Can also do encoding/newline conversion via -c option, as iconv can't do BOM or
newlines, and sometimes you just want "MS utf-8 mode" (`repr -c utf-8-sig+r`).
Using that with +i flag as e.g. `repr -c utf-8-sig+ri file1 file2 ...`
converts encoding+newlines+BOM for files in-place at no extra hassle.

##### [color](color)

Outputs terminal color sequences, making important output more distinctive.

Also can be used to interleave "tail -f" of several logfiles in the same terminal:

``` console
% t -f /var/log/app1.log | color red - &
% t -f /var/log/app2.log | color green - &
% t -f /var/log/app2.log | color blue - &
```

Or to get color-escape-magic for your bash script: `color red bold p`

##### [resolve-hostnames](resolve-hostnames)

Script (py3) to find all specified (either directly, or by regexp)
hostnames and replace these with corresponding IP addresses, resolved
through getaddrinfo(3).

Examples:

% cat cjdroute.conf
... "fraggod.net:21987": { ... },
"localhost:21987": { ... },
"fraggod.net:12345": { ... }, ...

% resolve-hostnames fraggod.net localhost < cjdroute.conf
... "192.168.0.11:21987": { ... },
"127.0.0.1:21987": { ... },
"192.168.0.11:12345": { ... }, ...

% resolve-hostnames -m '"(?P[\w.]+):\d+"' < cjdroute.conf
% resolve-hostnames fraggod.net:12345 < cjdroute.conf
% resolve-hostnames -a inet6 fraggod.net localhost < cjdroute.conf
...

% cat nftables.conf
define set.gw.ipv4 = { !ipv4.name1.local, !ipv4.name2.local }
define set.gw.ipv6 = { !ipv6.name1.local, !ipv6.name2.local }
...
# Will crash nft-0.6 because it treats names in anonymous sets as AF_INET (ipv4 only)

% resolve-hostnames -rum '!(\S+\.local)\b' -f nftables.conf
define set.gw.ipv4 = { 10.12.34.1, 10.12.34.2 }
define set.gw.ipv6 = { fd04::1, fd04::2 }
...

Useful a as conf-file pre-processor for tools that cannot handle names properly
(e.g. introduce ambiguity, can't deal with ipv4/ipv6, use weird resolvers, do it
dynamically, etc) or should not be allowed to handle these, convert lists of
names (in some arbitrary format) to IP addresses, and such.

Has all sorts of failure-handling and getaddrinfo-control cli options, can
resolve port/protocol names as well.

##### [resolve-conf](resolve-conf)

Python/Jinja2 script to produce a text file from a template, focused
specifically on templating configuration files, somewhat similar to
"resolve-hostnames" above or templating provided by ansible/saltstack.

Jinja2 env for template has following filters and values:

- `dns(host [, af, proto, sock, default, force_unique=True])` filter/global.

getaddrinfo(3) wrapper to resolve `host` (name or address) with optional
parameters to a single address, raising exception if it's non-unique by default.

af/proto/sock values can be either enum value names
(without AF/SOL/SOCK prefix) or integers.

- `hosts` - /etc/hosts as a mapping.

For example, hosts-file line `1.2.3.4 sub.host.example.org` will produce
following mapping (represented as yaml):

``` yaml
sub.host.example.org: 1.2.3.4
host.example.org:
sub: 1.2.3.4
org:
example:
host:
sub: 1.2.3.4
```

Can be used as a reliable dns/network-independent names. `--hosts-opts`
cli option allows some tweaks wrt how that file is parsed.
See also HostsNode object for various helper methods to lookup those.

- `iface` - current network interfaces and IPv4/IPv6 addresses
assigned there (fetched from libc getifaddrs via ctypes).

Example value structure (as yaml):

``` yaml
enp1s0:
- 10.0.0.134
- fd00::134
- 2001:470:1f0b:11de::134
- fe80::c646:19ff:fe64:632f
enp2s7:
- 10.0.1.1
lo:
- 127.0.0.1
- ::1
ip_vti0: []
```

Probably a good idea to use this stuff only when IPs are static and
get assigned strictly before templating.

- `❴% comment_out_if value[, comment-prefix] %❵...❴% comment_out_end %❵`

(curly-braces are weird to avoid jinja2 in github-pages - replace with normal ones)

Custom template block to prefix each non-empty line within it with specified
string (defaults to "#") if value is not false-y.

Can be used when format doesn't have block comments, but it's still desirable
to keep disabled things in dst file (e.g. for manual tinkering) instead of
using if-blocks around these, or to make specific lines easier to uncomment manually.

- `it` - itertools, `zip` builtin, `szip` - `zip(a.split(), b.split())`,
`_v`/`v_`/`_v_` - global funcs for adding spaces before/after/around non-empty strings.

- Whatever is loaded from `--conf-file/--conf-dir` (JSON/YAML files), if specified.

Use-case is a simple config file pre-processor for autonomous templating on service
startup with a minimal toolbox on top of jinja2, without huge dep-tree or any other
requirements and complexity, that is not scary to run from `ExecStartPre=` line as root.

##### [temp-patch](temp-patch)

Tool to temporarily modify (patch) a file - until reboot or for a specified
amount of time. Uses bind-mounts from tmpfs to make sure file will be reverted
to the original state eventually.

Useful to e.g. patch `/etc/hosts` with (pre-defined) stuff from LAN on a
laptop (so this changes will be reverted on reboot), or a notification filter
file for a short "busy!" time period (with a time limit, so it'll auto-revert
after), or stuff like that.

Even though dst file is mounted with "-o ro" by default (there's "-w" option to
disable that), linux doesn't seem to care about that option and mounts the thing
as "rw" anyway, so "chmod a-w" gets run on temp file instead to prevent
accidental modification (that can be lost).

There're also "-t" and "-m" flags to control timestamps during the whole
process.

##### [term-pipe](term-pipe)

Python script with various terminal input/output piping helpers and tools.

Has multiple modes for different use-cases, collected in same script mostly
because they're pretty simple and not worth remembering separate ones.

**out-paste**

Disables terminal echo and outputs line-buffered stdin to stdout.

Example use-case can be grepping through huge multiline strings
(e.g. webpage source) pasted into terminal, i.e.:

``` console
% term-pipe | g -o '\/tmp/errors.log" can be added at the end.

Check options of this subcommand for rate-limiting and some other tweaks.

##### [yaml-to-pretty-json](yaml-to-pretty-json)

Converts yaml files to an indented json, which is a bit more readable and
editable by hand than the usual compact one-liner serialization.

Due to yaml itself being json superset, can be used to convert json to
pretty-json as well.

##### [yaml-flatten](yaml-flatten)

Converts yaml/json files to a flat "key: value" lines.

Nested keys are flattened to a dot-separated "level1.level2.level3" keys,
replacing dots, spaces and colons there, to avoid confusing level separators
with the keys themselves.

Values are also processed to always be one-liners, handling long values
and empty lists/dicts and such in a readable manner too.

Output is intended for a human reader, to easily see value paths and such,
and definitely can't be converted back to yaml or any kind of data safely.

##### [yaml-diff](yaml-diff)

Tool to normalize YAML files' ordering/formatting and run "git diff | [delta]"
on those to produce nicely-colorized and useful diffs to inspect in the terminal.

Long YAMLs can be ordered and formatted in wildly different ways, and they often
are, when produced by different tools or edited manually, hence the need for
something to reformat them before running diff tools.

Script can be run on two dirs to compare all yml/yaml files in those recursively
(like "diff -r"), ignoring all other non-yaml files in there, as well as two
specific files.

Also has -f/--reformat option to pretty-print/normalize file(s) without diff,
which can be used to YAML-pretty-print JSON file(s) as well (incl. recursively,
with --fn-re override to match them). Requires python [pygments] module to be
installed for colorizing YAMLs printed to stdout with this option.

"git diff" can be used without "delta" if --no-delta option is set,
using its own colors (as per gitconfig), but output from [delta] is usually nicer,
has line numbers and highlights inline diffs.

Binaries and opts to both "git diff" and "delta" tools can be controlled
via env variables printed in -h/--help output.

[delta]: https://github.com/dandavison/delta
[pygments]: https://pygments.org/

##### [hz](hz)

Same thing as the common "head", but works with \\x00
(aka null char/byte , NUL, ␀, \\0, \\z, \\000, \\u0000, %00, ^@) delimeters.

Can be done with putting "tr" in the pipeline before and after "head",
but this one is maybe a bit less fugly.

Allows replacing input null-bytes with newlines in the output
(--replace-with-newlines option) and vice-versa.

Common use-case is probably has something to do with filenames and xargs, e.g.:

I have "h" as an alias for "head" in shells, so "head -z" (if there were such
option) would be aliased neatly to "hz", hence the script name.

Defaults to reading ALL lines, not just arbitrary number (like 10, which is
default for regular "head")!

##### [liac](liac)

"Log Interleaver And Colorizer" python script.

![interleaved colorized output][]

Reads lines from multiple files, ordering them by the specified field in the
output (default - first field, e.g. ISO8601 timestamp) and outputs each with
(optional) unique-filename-part prefix and unique (ansi-terminal, per-file) color.

Most useful for figuring out sequence of events from multiple timestamped logs.

To have safely-rotated logs with nice timestamps from any arbitrary command's
output, something like `stdbuf -oL | svlogd -r _ -ttt `
can be used.

Note "stdbuf" coreutils tool, used there to tweak output buffering, which usually
breaks such timestamps, and "svlogd" from [runit] suite (no deps, can be built separately).

See [blog post about liac tool] for more info.

[interleaved colorized output]:
https://blog.fraggod.net/images/liac_interleaved_colorized_output.jpg
[runit]: https://smarden.org/runit/
[blog post about liac tool]:
https://blog.fraggod.net/2015/12/29/tool-to-interleave-and-colorize-lines-from-multiple-log-or-any-other-files.html

##### [html-embed](html-embed)

Script to create "fat" HTML files, embedding all linked images
(as base64-encoded data-urls), stylesheets and js into them.

All src= and href= paths must be local (e.g. "js/script.js" or "/css/main.css"),
and will simply be treated as path components (stripping slashes on the left)
from html dir, nothing external (e.g. "//site.com/stuff.js") will be fetched.

Doesn't need anything but python, based on stdlib html.parser module.

Not optimized for huge amounts of embedded data, storing all the substitutions
in memory while it runs, and is unsafe to run on random html files, as it can
embed something sensitive (e.g. ``) - no extra checks there.

Use-case is to easily produce single-file webapps or pages to pass around (or
share somewhere), e.g. some d3-based interactive chart page or an html report
with a few embedded images.

##### [someml-indent](someml-indent)

Simple and dirty regexp + backreferences something-ML (SGML/HTML/XML) parser to
indent tags/values in a compact way without messing-up anything else in there.

I.e. non-closed tags are FINE, something like <@> doesn't cause parser to
explode, etc.

Does not add any XML headers, does not mangle (or "canonize") tags/attrs/values
in any way, except for stripping/adding those spaces.

Kinda like BeautifulSoup, except not limited to html and trivial enough so that
it can be trusted not to do anything unnecessary like stuff mentioned above.

For cases when `xmllint --format` fail and/or break such kinda-ML-but-not-XML files.

##### [hashname](hashname)

Script to add simple/distinctive base32-encoded content hash to filenames.

For example:

``` console
% hashnames -p *.jpg

wallpaper001.jpg -> wallpaper001.kw30e7cqytmmw.jpg
wallpaper893.jpg -> wallpaper893.vbf0t0qht4dd0.jpg
wallpaper895.jpg -> wallpaper895.q5mp0j95bxbdr.jpg
wallpaper898.jpg -> wallpaper898.c9g9yeb06pdbj.jpg
```

For collecting files with commonly-repeated names into some dir,
like random "wallpaper.jpg" or "image.jpg" images above from the internets.

Can also be used with -t/--tag option to update names for changed files,
which is handy in web-accessible dirs for changing URLs to invalidate caches.

Use -h/--help for info on more useful options.

##### [hhash](hhash.ml)

Produces lower-entropy "human hash" phrase consisting of aspell english
dictionary words for input arg(s) or data on stdin.

It works by first calculating BLAKE2 hash of input string/data via [libsodium],
and then encoding it using consistent word-alphabet, exactly like something
like base32 or base64 does.

Example:

``` console
% hhash -e AAAAC3NzaC1lZDI1NTE5AAAAIPh5/VmxDwgtJI0HiFBqZkbyV1I1YK+2DVjGjYydNp5o
allan avenues regrade windups flours
entropy-stats: word-count=5 dict-words=126643 word-bits=17.0 total-bits=84.8
```

Here -e is used to print entropy estimate for produced words.

Note that resulting entropy values can be fractional if word-alphabet ends up
being padded to map exactly to N bits (e.g. 17 bits above), so that words in it
can be repeated, hence not exactly 17 bits of distinct values.

Written in OCAML, linked against [libsodium] (for BLAKE2 hash function)
via small C glue code. Build it with:

``` console
% ocamlopt -o hhash -O2 unix.cmxa str.cmxa \
-cclib -lsodium -ccopt -Wl,--no-as-needed hhash.ml hhash.ml.c
% strip hhash
```

Caches dictionary into a ~/.cache/hhash.dict (-c option) on first run to produce
consistent results on this machine. Updating that dictionary will change outputs!

[libsodium]: https://libsodium.org/

##### [crypt](crypt)

Trivial file/stream encryption tool using [PyNaCl's]
crypto_secretstream_xchacha20poly1305 authenticated encryption API.

Key can be either specified on the command line for simplicity or read from a
file, and is always processed via scrypt, as it's likely some short string.

Usage examples:

``` console
% crypt -ek my-secret-key secret.tar secret.tar.enc
% crypt -dk my-secret-key secret.tar.enc secret.tar.test
% crypt -ek @~/.secret.key secret.tar.enc
```

Intended for an ad-hoc temporary encryption when transferring stuff via a usb
stick, making a temporary backup to a random untrusted disk or whatever.

Does not support any kind of appending/resuming or partial operation, which can
be bad if there's a flipped bit anywhere in the encrypted data - decryption will
stop and throw error at that point.

[PyNaCl's]: https://pynacl.readthedocs.io/

#### Kernel sources/build/version management

##### [kernel-patch](kernel-patch)

Simple stateless script to update sources in /usr/src/linux to some (specified)
stable version.

Looks for "patch-X.Y.Z.xz" files (as provided on kernel.org) under
/usr/src/distfiles (configurable at the top of the script), or downloads them
there from kernel.org.

Does update (or rollback) by grabbing current patchset version from Makefile and
doing essentially `patch -R < && patch < ` - i.e.
rolling-back the current patchset, then applying new patch.

Always does `patch --dry-run` first to make sure there will be no mess left
over by the tool and updates will be all-or-nothing.

In short, allows to run e.g. `kernel-patch 3.14.22` to get 3.14.22 in
`/usr/src/linux` from any other clean 3.14.\* version, or just
`kernel-patch` to have the latest 3.14 patchset.

##### [kernel-conf-check](kernel-conf-check)

Ad-hoc python script to check any random snippet with linux kernel
`CONFIG_...` values (e.g. "this is stuff you want to set" block on some wiki)
against kernel config file, current config in /proc/config.gz or such.

Reports what matches and what doesn't to stdout, trivial regexp matching.

##### [clean-boot](clean-boot)

Script to remove older kernel versions (as installed by `/sbin/installkernel`)
from `/boot` or similar dir.

Always keeps version linked as "vmlinuz", and prioritizes removal of older
patchset versions from each major one, and only then latest per-major patchset,
until free space goal (specified percentage, 20% by default) is met.

Also keeps specified number of last-to-remove versions, can prioritize cleanup
of ".old" verssion variants, keep `config-*` files... and other stuff (see --help).

Example:

# clean-boot --debug --dry-run -f 100
DEBUG:root:Preserved versions (linked version, its ".old" variant, --keep-min): 4
DEBUG:root: - 3.9.9.1 - System.map-3.9.9-fg.mf_master
DEBUG:root: - 3.9.9.1 - config-3.9.9-fg.mf_master
DEBUG:root: - 3.9.9.1 - vmlinuz-3.9.9-fg.mf_master
DEBUG:root: - 3.10.27.1 - vmlinuz-3.10.27-fg.mf_master
...
DEBUG:root: - 3.12.19.1 - System.map-3.12.19-fg.mf_master
DEBUG:root: - 3.12.20.1 - config-3.12.20-fg.mf_master
DEBUG:root: - 3.12.20.1 - System.map-3.12.20-fg.mf_master
DEBUG:root: - 3.12.20.1 - vmlinuz-3.12.20-fg.mf_master
DEBUG:root:Removing files for version (df: 58.9%): 3.2.0.1
DEBUG:root: - System.map-3.2.0-fg.mf_master
DEBUG:root: - config-3.2.0-fg.mf_master
DEBUG:root: - vmlinuz-3.2.0-fg.mf_master
DEBUG:root:Removing files for version (df: 58.9%): 3.2.1.0
... (removal of older patchsets for each major version, 3.2 - 3.12)
DEBUG:root:Removing files for version (df: 58.9%): 3.12.18.1
... (this was the last non-latest patchset-per-major)
DEBUG:root:Removing files for version (df: 58.9%): 3.2.16.1
... (removing latest patchset for each major version, starting from oldest - 3.2 here)
DEBUG:root:Removing files for version (df: 58.9%): 3.7.9.1
...
DEBUG:root:Removing files for version (df: 58.9%): 3.8.11.1
...
DEBUG:root:Finished (df: 58.9%, versions left: 4, versions removed: 66).

("df" doesn't rise here because of --dry-run, `-f 100` =
"remove all non-preserved" - as df can't really get to 100%)

Note how 3.2.0.1 (non-.old 3.2.0) gets removed first, then 3.2.1, 3.2.2, and so
on, but 3.2.16 (latest of 3.2.X) gets removed towards the very end, among other
"latest patchset for major" versions, except those that are preserved unconditionally
(listed at the top).

#### ZNC log helpers

Couple scripts to manage [ZNC IRC bouncer](https://znc.in/) logs -
archive, view, search, etc.

##### [znc-log-aggregator](znc-log-aggregator)

Tool to process ZNC chat logs, produced by "log" module (one enabled globally,
with default wildcards) and store them using following schema under some -d/--log-dir:

/chat/__-.log.xz
/priv/__-.log.xz

Where "priv" differs from "chat" in latter being prefixed by "#" or "&".

With values from ZNC log paths: `moddata/log/*///.log`

Each ZNC-log line gets processed by regexp to add proper date, so that one
doesn't have to use long timestamps in ZNC itself:
`[HH:MM:SS] some msg` -> `[yy-mm-dd HH:MM:SS] some msg`.

Latest (current day) logs are skipped.
New logs for each run are concatenated into a monthly .xz file.

Should be safe to stop at any time without data loss -
all resulting .xz's get written to temporary files and renamed at the very end,
followed by unlinking of the source files, with nothing changed or updated in-place.

All temp files are produced in the destination dir, even with --dry-run,
and should be cleaned-up on any abort/exit/finish.

Idea is to have more convenient hierarchy and less files for easier shell
navigation/grepping (xzless/xzgrep), and without needing to worry about space
usage of uncompressed logs in the long run.

ZNC changed how it stores logs a few times over the years, and this tools
also helped maintain consistent storage schema across these.

##### [znc-log-reader](znc-log-reader)

Same as znc-log-aggregator above, but seeks/reads specific tail ("last n lines")
or time range (with additional filtering by channel/nick and network) from all
current and aggregated (via that aggregator script) ZNC logs.

Mostly used to query/grep recent chat logs by approximate channel name from terminal easily.

#### systemd-related

##### [systemd-dashboard]

[systemd-dashboard]: systemd-dashboard

Python script to list all currently active and non-transient systemd units,
so that these can be tracked as a "system state",
and e.g. any deviations there detected/reported (simple diff can do it).

Gets unit info by parsing Dump() snapshot fetched via sd-bus API of libsystemd
(using ctypes to wrap it), which is same as e.g. "systemd-analyze dump" gets.

Has -m/--machines option to query state from all registered machines as well,
which requires root (for sd_bus_open_system_machine) due to current systemd limitations.

See [Dashboard-for-... blog post] for extended rationale,
though it's probably obsolete otherwise since this thing was rewritten.

[Dashboard-for-... blog post]:
https://blog.fraggod.net/2011/2/Dashboard-for-enabled-services-in-systemd

##### [systemd-watchdog](systemd-watchdog)

Trivial script to ping systemd watchdog and do some trivial actions in-between
to make sure os still works.

Wrote it after yet another silent non-crash, where linux kernel refuses to
create new pids (with some backtraces) and seem to hang on some fs ops, blocking
syslog/journal, but leaving most simple daemons running ok-ish for a while.

So this trivial script, tied into systemd-controlled watchdog timers, tries to
create pids every once in a while, with either hang or crash bubbling-up to
systemd (pid-1), which should reliably reboot/crash the system via hardware wdt.

Example watchdog.service:

``` ini
[Service]
Type=notify
ExecStart=/usr/local/bin/systemd-watchdog -i30 -n \
-f /var/log/wdt-fail.log \
-x 'ip link' -x 'ip addr' -x 'ip ro' -x 'journalctl -an30'

WatchdogSec=60
TimeoutStartSec=15
Restart=on-failure
RestartSec=20
StartLimitInterval=10min
StartLimitBurst=5
StartLimitAction=reboot-force

[Install]
WantedBy=multi-user.target
```

(be sure to tweak timeouts and test without "reboot-force" first though,
e.g. pick RestartSec= for transient failures to not trigger StartLimitAction)

Can optionally get IP of (non-local) gateway to 1.1.1.1 (or any specified IPv4)
via libmnl (also used by iproute2, so always available) and check whether it
responds to [fping] probes, crashing if it does not - see `-n/--check-net-gw` option.

That's mainly for remote systems which can become unreachable if kernel network
stack, local firewall, dhcp, ethernet or whatever other link fails (usually due
to some kind of local tinkering), ignoring more mundane internet failures.

To avoid reboot loops (in abscence of any networking), it might be a good idea
to only start script with this option manually (e.g. right before messing up
with the network, or on first successful access).

`-f/--fail-log` option is to log date/time of any failures for latest boot
and run `-x/--fail-log-cmd` command(s) on any python exceptions (note: kernel
hangs probably won't cause these), logging their stdout/stderr there -
e.g. to dump network configuration info as in example above.

Useless without systemd and requires systemd python module, plus fping tool
if `-n/--check-net-gw` option is used.

[fping]: https://fping.org/

##### [cgrc](cgrc)

Wrapper for [systemd.resource control] stuff to run commands in transient
scopes within pre-defined slices, as well as wait for these and list pids
within them easily.

Replacement for things like libcgroup, cgmanager and my earlier
[cgroup-tools project], compatible with [unified cgroup-v2 hierarchy]
and working on top of systemd (use `systemd.unified_cgroup_hierarchy`
on cmdline, if non-default).

Resource limits for cgrc scopes should be defined via hierarchical slices like these:

``` ini
# apps.slice
[Slice]

CPUWeight=30
IOWeight=30

MemoryHigh=5G
MemoryMax=8G
MemorySwapMax=1G

# apps-browser.slice
[Slice]
CPUWeight=30
IOWeight=30
MemoryHigh=3G
```

And then script can be used to start things there:

``` console
% cgrc apps-browser -- chromium
% cgrc -u ff apps-browser -- firefox --profile myprofile
```

Where e.g. last command would end up running something like this:

```console
% systemd-run -q --user --scope --unit ff \
--slice apps-browser -- firefox --profile myprofile
```

Note that .scope cgroups are always transient (vanish after run), and only
.slice ones can be pre-defined with limits.
Both get started/stopped by systemd on as-needed basis.

Tool also allows to check or list pids within scopes/slices with -c/-l options
(to e.g. check if named scope already started or something running in a slice),
as well as waiting on these (-q option, can be used to queue/run commands in sequence)
and manipulating associated cgroup limits easily (-v option).

Run without any args/opts or with -h/--help to get more detailed usage info.

[cgroup-tools project]: https://github.com/mk-fg/cgroup-tools
[unified cgroup-v2 hierarchy]: https://www.kernel.org/doc/Documentation/cgroup-v2.txt
[systemd.resource control]: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html

#### SSH and WireGuard related

See also: [autossh] and such.

[autossh]: https://www.harding.motd.ca/autossh/

##### [mosh-nat] / [mosh-nat-bind.c]
[mosh-nat]: mosh-nat
[mosh-nat-bind.c]: mosh-nat-bind.c

Python wrapper for mosh-server binary to do UDP hole punching through
local NAT setup before starting it.

Comes with mosh-nat-bind.c source for LD_PRELOAD=./mnb.so lib to force
mosh-client on the other side to use specific local port that was used in
"mosh-nat".

Example usage (server at 84.217.173.225, client at 74.59.38.152):

``` console
server% ./mosh-nat 74.59.38.152
mosh-client command:
MNB_PORT=34730 LD_PRELOAD=./mnb.so
MOSH_KEY=rYt2QFJapgKN5GUqKJH2NQ mosh-client 34730

client% MNB_PORT=34730 LD_PRELOAD=./mnb.so \
MOSH_KEY=rYt2QFJapgKN5GUqKJH2NQ mosh-client 84.217.173.225 34730
```

Notes:

- mnb.so is mosh-nat-bind.c lib. Check its header for command to build it.
- Both mnb.so and mosh-nat only work with IPv4, IPv6 shouldn't use NAT anyway.
- Should only work like that when NAT on either side doesn't rewrite src ports.
- 34730 is default for `-c/--client-port` and `-s/--server-port` opts.
- Started mosh-server waits for 60s (default) for mosh-client to connect.
- Continous operation relies on mosh keepalive packets without interruption.
- No roaming of any kind is possible here.
- New MOSH_KEY is generated by mosh-server on every run.

Useful for direct and fast connection when there's some other means
of access available already, e.g. ssh through some slow/indirect tunnel
or port forwarding setup.

For more hands-off hole-punching, similar approach to what [pwnat] does can be used.\
See [mobile-shell/mosh#623] for more info and links on such feature implemented in mosh directly.\
Source for LD_PRELOAD lib is based on

[pwnat]: https://samy.pl/pwnat/
[mobile-shell/mosh#623]: https://github.com/mobile-shell/mosh/issues/623

##### [tping](tping)

Python (asyncio) tool to try connecting to specified TCP port until connection
can be established, then just exit, i.e. to wait until some remote port is accessible.

Can be used to wait for host to reboot before trying to ssh into it, e.g.:

% tping myhost && ssh root@myhost

(default `-p/--port` is 22 - ssh, see also `-s/--ssh` option)

Tries establishing new connection (forcing new SYN, IPv4/IPv6 should both work)
every `-r/--retry-delay` seconds (default: 1), only discarding (closing) "in
progress" connections after `-t/--timeout` seconds (default: 3), essentially
keeping rotating pool of establishing connections until one of them succeeds.

This means that with e.g. `-r1 -t5` there will be 5 establishing connections
(to account for slow-to-respond remote hosts) rotating every second, so ratio of
these delays shouldn't be too high to avoid spawning too many connections.

Host/port names specified on the command line are resolved synchronously on
script startup (same as with e.g. "ping" tool), so it can't be used to wait
until hostname resolves, only for connection itself.

Above example can also be shortened via `-s/--ssh` option, e.g.:

% tping -s myhost 1234
% tping -s root@myhost:1234 # same thing as above
% tping -s -p1234 myhost # same thing as above

Will exec `ssh -p1234 root@myhost` immediately after successful tcp connection.

Uses python stdlib stuff, namely asyncio, to juggle multiple connections
in an efficient manner.

#### WiFi / Bluetooth helpers

##### [adhocapd](adhocapd)

Picks first wireless dev from `iw dev` and runs [hostapd] + udhcpd
(from busybox) on it.

Configuration for both is generated using reasonable defaults - distinctive
(picked from `ssid_list` at the top of the script) AP name and
random password (using `passgen` from this repo or falling back to
`tr -cd '[:alnum:]'
##### [wpa-systemd-wrapper](wpa-systemd-wrapper)

Systemd wrapper for [wpa_supplicant] or [hostapd], enabling either to
work with Type=notify, support WatchdogSec=, different exit codes and
all that goodness.

Starts the daemon as a subprocess, connecting to its management interface and
watching state/wpa_state changes, only indicating "started" state for systemd
when daemon actually starts scanning/connecting (for wpa_supplicant) or sets
state=enabled for hostapd.

WatchdogSec= issues PING commands to underlying daemon, proxying responses back,
as long as daemon state is somehting valid, and not INTERFACE-DISABLED,
locally-generated disconnect or such, usually indicating hw failure, kernel
module issue or whatever else.

Such thing is needed to have systemd unit state follow AP/STA state, failing
when e.g. wifi dongle gets pulled out from USB port, as that doesn't actually
cause these things to fail/exit otherwise, which might be desirable if that wifi
link is critical to other services or as a reboot-workaround for driver bugs.

Example systemd unit (AP mode):

``` ini
[Service]
ExecStart=/usr/local/bin/wpa-systemd-wrapper \
--exit-check '/run/wpa.wlan0.first-run:config' \
--ap-mode wlan0 /etc/hostapd.wlan0.conf

Type=notify
WatchdogSec=90
Restart=on-failure
RestartPreventExitStatus=78
RestartSec=3
# StartLimitInterval=8min
# StartLimitBurst=10
# StartLimitAction=reboot
```

This will run hostapd (due to `-a/--ap-mode`), and exit with special 78/CONFIG
code if "first-run" file exists and hostapd never gets into ENABLED state on the
first attempt - i.e. something likely wrong with the config and there's no point
restarting it ad nauseum.

Python/asyncio, requires python-systemd installed, use `-h/--help`
and `-d/--debug` opts for more info.

[wpa_supplicant]: https://w1.fi/wpa_supplicant/

##### [timed-ble-beacon](timed-ble-beacon)

Python script to broadcast [Bluetooth Low Energy (BLE)] beacons
for specified amount of time, with a time countdown in them,
using standard linux [BlueZ bluetooth stack].

Broadcasts are done using Primary Advertising mechanism (ADV\_SCAN\_IND PDUs),
not marked as "discoverable", intended be picked-up by passive scans on recipient.
All data is embedded in "Manufacturer Specific Data" bytes, where in addition to
countdown, there's also replay counter and keyed HMAC, to prevent replays or a
simple forgery.

Both sender (broadcaster) and recipient (observer) should share configured keys
for communication to work.

Intended to be used to temporarily enable/disable something while BLE beacons
are being broadcast, receiving/checking those on cheap [micropython] controllers -
kinda like a smart-home remotely-controlled switch, but automatically reverting
to default state on its own, standalone, and much simpler.

[timed-ble-beacon-mpy-led micropython script] is the receiver side,
intended to run on a cheap [RPi Pico W] board with rp2040 microcontroller.
There can be any number of senders/receivers at the same time - just use
different `--mid` and `-s/--secret` values for different control-domains.

Can be debug-run like this: `./timed-ble-beacon -dt 5m`

Uses [python-dbus] and [python-gobject] modules to work
with bluez over dbus within glib eventloop.

[BlueZ bluetooth stack]: https://www.bluez.org/
[Bluetooth Low Energy (BLE)]: https://en.wikipedia.org/wiki/Bluetooth_Low_Energy
[micropython]: https://docs.micropython.org/en/latest/
[RPi Pico W]: https://www.raspberrypi.com/documentation/microcontrollers/pico-series.html#picow-technical-specification
[timed-ble-beacon-mpy-led micropython script]: #hdr-timed-ble-beacon-mpy-led
[python-dbus]: https://dbus.freedesktop.org/doc/dbus-python/
[python-gobject]: https://pygobject.gnome.org/

##### [timed-ble-beacon-mpy-led](timed-ble-beacon-mpy-led)

Micopython script to passively scan for [Bluetooth Low Energy (BLE)] beacons
with specific HMAC-authenticated payload, and trigger some action while those
are active, reverting back to default state otherwise.

Intended to run on [RPi Pico W] or [ESP32] microcontrollers, or anything else
supported by [micropython], and default code just blinks connected LEDs in a
configured pattern, as an indicator/notification task.

[timed-ble-beacon script] above can be used to broadcast BLE beacons in question.
Must be configured with at least mid/key parameters when calling it via import +
run_with_conf() or at the top of the script, unless just testing with defaults
in both of these.

To setup/run this on a usb-tty-connected microcontroller board:

``` console
## Upload micropython firmware to the device, install "mpremote" tool

% nano timed-ble-beacon-mpy-led
## Set parameters like ble_mid and ble_secret at the top of the script

% mpremote run timed-ble-beacon-mpy-led
## Should either work or print some errors to console

## To setup this script to run on board boot with all-hardcoded parameters
% mpremote cp timed-ble-beacon-mpy-led :main.py
% mpremote reset

## Alternatively, configuration can be provided via loader-script
% mpremote cp timed-ble-beacon-mpy-led :tbbml.py
% echo >main.py 'import tbbml'
% echo >>main.py 'tbbml.run_with_conf(verbose=1, led_pin=2, ble_mid=123, ble_secret=b"test")'
% mpremote cp main.py :

## Can also compile "tbbml" for loader-script via mpy-cross for size/mem/load-times
% mpy-cross -march=armv6m -O2 timed-ble-beacon-mpy-led -o tbbml.mpy
% mpremote cp tbbml.mpy :
```

Action-task in this script simply blinks LED indicator (e.g. built-in
`machine.Pin('LED')` by default, can use multiple LEDs) with randomized
intervals when no beacons are detected.

See Conf class in this, as well as timed-ble-beacon script above
and its `-h/--help` output for more details.

[ESP32]: https://en.wikipedia.org/wiki/ESP32
[timed-ble-beacon script]: #hdr-timed-ble-beacon

#### Misc

Misc one-off scripts that don't group well with anythin else.

##### [at](at)

Replacement for standard unix'ish "atd" daemon in the form of a bash script.

It just forks out and waits for however long it needs before executing the given command.\
Unlike atd proper, such tasks won't survive reboot, obviously.

Usage: ./at [ -h | -v ] when < sh_script
With -v flag ./at mails script output if it's not empty even if exit code is zero.

##### [sleepc](sleepc)

Python script that works like a verbose "sleep" tool - prints countdown until
specified time to terminal, and also parses more wide variety of relative/absolute
timestamp formats:

``` console
% ./sleepc 3h2m
Parsed time-spec '3h2m' as 2023-06-11 23:10:12.459720 [in 3h 2m]
Countdown: 3:01:59 [in 3h 2m]
```

Useful for waiting with a known time or delay in interactive consoles, to avoid
needing to calculate offset for "sleep", and be able to check back on it later.

##### [wgets](wgets)

Simple script to grab a file using wget and then validate checksum of
the result, e.g.:

``` console
$ wgets -c https://os.archlinuxarm.org/os/ArchLinuxARM-sun4i-latest.tar.gz cea5d785df19151806aa5ac3a917e41c

Using hash: md5
Using output filename: ArchLinuxARM-sun4i-latest.tar.gz
--2014-09-27 00:04:45-- https://os.archlinuxarm.org/os/ArchLinuxARM-sun4i-latest.tar.gz
Resolving os.archlinuxarm.org (os.archlinuxarm.org)... 142.4.223.96, 67.23.118.182, 54.203.244.41, ...
Connecting to os.archlinuxarm.org (os.archlinuxarm.org)|142.4.223.96|:80... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

The file is already fully retrieved; nothing to do.

Checksum matched
```

Basic invocation syntax is `wgets [ wget_opts ] url checksum`,
checksum is hex-decoded and hash func is auto-detected from its length
(md5, sha-1, all sha-2's are supported).

Idea is that - upon encountering an http link with either checksum on the page
or in the file nearby - you can easily run the thing providing both link and
checksum to fetch the file.

If checksum is available in e.g. \*.sha1 file alongside the original one,
it might be a good idea to fetch that checksum on a different host or a proxy,
making spoofing of both checksum and the original file on the same connection
a bit harder.

##### [mail](mail)

Simple bash wrapper for sendmail command, generating From/Date headers and
stuff, just like mailx would do, but also allowing to pass custom headers
(useful for filtering error reports by-source), which some implementations
of "mail" fail to do.

##### [passgen](passgen)

Uses aspell english dictionaly to generate easy-to-remember passphrase -
a [Diceware-like] method.

[Diceware-like]: https://en.wikipedia.org/wiki/Diceware

Use -e option to get a rough entropy estimate for the resulting passphrase,
based on number of words in aspell dictionary dump that is being used.

Other options allow for picking number of words and sanity-checks like min/max
length (to avoid making it too unwieldy or easy to bruteforce via other methods).

##### [urlparse](urlparse)

Simple script to parse long URL with lots of parameters, decode and print it out
in an easily readable ordered YAML format or diff (that is, just using "diff"
command on two outputs) with another URL.

No more squinting at some huge incomprehensible ecommerce URLs before scraping
the hell out of them!

##### [ip-ext](ip-ext)

Some minor tools for network configuration from console/scripts, which iproute2
seem to be lacking, in a py3 script.

For instance, if network interface on a remote machine was (mis-)configured in
initramfs or wherever to not have link-local IPv6 address, there seem to be no
tool to restore it without whole "ip link down && ip link up" dance, which can
be a bad idea.

`ipv6-lladdr` subcommand handles that particular case, generating ipv6-lladdr
from mac, as per RFC 4291 (as implemented in "netaddr" module) and can assign
resulting address to the interface, if missing:

``` console
# ip-ext --debug ipv6-lladdr -i enp0s9 -x
DEBUG:root:Got lladdr from interface (enp0s9): 00:e0:4c:c2:78:86
DEBUG:root:Assigned ipv6_lladdr (fe80::2e0:4cff:fec2:7886) to interface: enp0s9
```

`ipv6-dns` tool generates \*.ip.arpa and djbdns records for specified IPv6.

`ipv6-name` encodes or hashes name into IPv6 address suffix to produce
an easy-to-remember static ones.

`iptables-flush` removes all iptables/ip6tables rules from all tables,
including any custom chains, using iptables-save/restore command-line tools,
and sets policy for default chains to ACCEPT.

##### [blinky](blinky)

Script to blink gpio-connected leds via `/sys/class/gpio` interface.

Includes oneshot mode, countdown mode (with some interval scaling option),
direct on-off phase delay control (see --pre, --post and --interval\* options),
cooperation between several instances using same gpio pin, "until" timestamp
spec, and generally everything I can think of being useful (mostly for use from
other scripts though).

##### [openssl-fingerprint](openssl-fingerprint)

Do `openssl s_client -connect somesite
##### [nsh](nsh)

Bash script to "nsenter" into specified machine's (as can be seen in
`ps -eo machine` or `nsh` when run without args) container namespaces
and run login shell there.

Machine in question must run systemd as pid-1 (e.g. systemd-nspawn container),
as it gets picked as `--target` pid for nsenter.

Very similar to `machinectl login `, but does not asks for
user/password and does not start new `systemd --user` session,
just runs `su -` to get root login shell.

Essentially same as `machinectl shell `, but doesn't require
systemd-225+ and machine being registered with systemd at all.

If running `tty` there says `not a tty` and e.g. `screen` bails out with
`Must be connected to a terminal.`, just run extra `getty tty` there - will
ask to login (be mindful of /etc/securetty if login fails), and everything
tty-related should work fine afterwards.

If run without argument or with `-l/--list` option, will list running machines.

See also: lsns(1), nsenter(1), unshare(1)

##### [pam-run](pam-run)

Wrapper that opens specified PAM session (as per one of the configs in
`/etc/pam.d`, e.g. "system-login"), switches to specified uid/gid and runs
some command there.

My use-case is to emulate proper "login" session for systemd-logind, which
neither "su" nor "sudo" can do (nor should do!) in default pam configurations
for them, as they don't load pam_systemd.so (as opposed to something like
`machinectl shell myuser@ -- ...`).

This script can load any pam stack however, so e.g. running it as:

# pam-run -s system-login -u myuser -t :1 \
-- bash -c 'systemctl --user import-environment \
&& systemctl --user start xorg.target && sleep infinity'

Should initiate proper systemd-logind session (and close it afterwards)
and start "xorg.target" in "myuser"-specific "systemd --user" instance
(started by logind with the session).

Can be used as a GDM-less way to start/keep such sessions (with proper
display/tty and class/type from env) without much hassle or other weirdness
like "agetty --autologin" or "login" in some pty (see also [mk-fg/de-setup] repo),
or for whatever other pam wrapping or testing (e.g. try logins with passwords
from file), as it has nothing specific (or even related) to desktops.

Self-contained python script, using libpam via ctypes.

Warning: this script is no replacement for su/sudo wrt uid/gid-switching, and
doesn't implement all the checks and sanitization these tools do, so only
intended to be run from static, clean or trusted environment (e.g. started by
systemd or manually).

[mk-fg/de-setup]: https://github.com/mk-fg/de-setup

##### [primes](primes)

Python script to print prime numbers in specified range.

For small ranges only, as it does dumbest brute-force \[2, sqrt(n)\] division checks,
and intended to generate primes for non-overlapping "tick % n" workload spacing,
not any kind of crypto operations.

##### [boot-patcher](boot-patcher)

Py script to run on early boot, checking specific directory for update-files
and unpack/run these, recording names to skip applied ones on subsequent boots.

Idea for it is to be very simple, straightforward, single-file drop-in script to
put on distributed .img files to avoid re-making these on every one-liner change,
sending tiny .update files instead.

Update-file format:

- Either zip or bash script with .update suffix.
- Script/zip detected by python's zipfile.is_zipfile() (zip file magic).
- If zip, should contain "_install" (update-install) script inside.
- Update-install script shebang is optional, defaults to "#!/bin/bash".

Update-install script env:

- BP_UPDATE_ID: name of the update (without .update suffix, e.g. "001.test").

- BP_UPDATE_DIR: unpacked update zip dir in tmpfs.

Will only have "_install" file in it for standalone scripts (non-zip).

- BP_UPDATE_STATE: /var/lib/boot-patcher/

Persistent dir created for this update, can be used to backup various
updated/removed files, just in case.
If left empty, removed after update-install script is done.

- BP_UPDATE_STATE_ROOT: /var/lib/boot-patcher

- BP_UPDATE_REBOOT: reboot-after flag-file (on tmpfs) to touch.

If reboot is required after this update, create (touch) file at that path.\
Reboot will be done immediately after this particular update, not after all of them.

- BP_UPDATE_REAPPLY: flag-file (on tmpfs) to re-run this update on next boot.

Can be used to retry failed updates by e.g. creating it at the start of the
script and removing on success.

Example update-file contents:

- 2017-10-27.001.install-stuff.zip.update

`_install`:

cd "$BP_UPDATE_DIR"
exec pacman --noconfirm -U *.pkg.tar.xz

`*.pkg.tar.xz` - any packages to install, zipped alongside that ^^^

- 2017-10-28.001.disable-console-logging.update (single update-install file):

patch -l /boot/boot.ini <<'EOF'
--- /boot/boot.ini.old 2017-10-28 04:11:15.836588509 +0000
+++ /boot/boot.ini 2017-10-28 04:11:38.000000000 +0000
@@ -6,7 +6,7 @@
hdmitx edid

setenv condev "console=ttyAML0,115200n8 console=tty0"
-setenv bootargs "root=/dev/mmcblk1p2 ... video=HDMI-A-1:1920x1080@60e"
+setenv bootargs "root=/dev/mmcblk1p2 ... video=HDMI-A-1:1920x1080@60e loglevel=1"

setenv loadaddr "0x1080000"
setenv dtb_loadaddr "0x1000000"
EOF
touch "$BP_UPDATE_REBOOT"

- 2017-10-28.002.apply-patches-from-git.zip.update

`_install`:

set -e -o pipefail
cd /srv/app
for p in "$BP_UPDATE_DIR"/*.patch ; do patch -p1 -i "$p"; done

`*.patch` - patches for "app" from the repo, made by e.g. `git format-patch -3`.

Misc notes:

- Update-install exit code is not checked.

- After update-install is finished, and if BP_UPDATE_REAPPLY was not created,
".done" file is created in BP_UPDATE_STATE_ROOT and update is
skipped on all subsequent runs.

- Update ordering is simple alphasort, dependenciess can be checked by update
scripts via .done files (also mentioned in prev item).

- No auth (e.g. signature checks) for update-files, so be sure to send these
over secure channels.

- Run as `boot-patcher --print-systemd-unit` for the only bit of setup it needs.

##### [audit-follow](audit-follow)

Simple py3 script to decode audit messages from "journalctl -af -o json" output,
i.e. stuff like this:

Jul 24 17:14:01 malediction audit: PROCTITLE
proctitle=7368002D630067726570202D652044... (loooong hex-encoded string)
Jul 24 17:14:01 malediction audit: SOCKADDR saddr=020000517F0000010000000000000000

Into this:

PROCTITLE proctitle='sh -c grep -e Dirty: -e Writeback: /proc/meminfo'
SOCKADDR saddr=127.0.0.1:81

Filters for audit messages only, strips long audit-id/time prefixes,
unless -a/--all specified, puts separators between multi-line audit reports,
relative and/or differential timestamps (-r/--reltime and -d/--difftime opts).

Audit subsystem can be very useful to understand which process modifies some
path, what's the command-line of some /bin/bash being run from somewhere
occasionally, or what process/command-line connects to some specific IP and what
scripts it opens beforehand - all without need for gdb/strace, or where they're
inapplicable.

Some useful incantations (cheatsheet):

# auditctl -e 1
# auditctl -a exit,always -S execve -F path=/bin/bash
# auditctl -a exit,always -F auid=1001 -S open -S openat
# auditctl -w /some/important/path/ -p rwxa
# auditctl -a exit,always -F arch=b64 -S connect

# audit-follow -ro='--since=-30min SYSLOG_IDENTIFIER=audit' |
grep --line-buffered -B1000 -F some-interesting-stuff | tee -a audit.log

# auditctl -e 0
# auditctl -D

auditd + ausearch can be used as an offline/advanced alternative to such script.\
More powerful options for such task on linux can be sysdig and various BPF tools.

##### [tui-binary-conv](tui-binary-conv)

Simple ncurses-based interactive (TUI) decimal/hex/binary
py3 converter script for the terminal.

Main purpose it to easily experiment with flipping bits and digits in values,
seeing nicely aligned/formatted/highlighted immediate changes in other outputs
and an easy converter tool as well.

Controls are: cursor keys, home/end, backspace, insert (insert/replace mode),
0/1 + digits + a-f, q to quit.

There's a picture of it [on the blog page here].

[on the blog page here]:
https://blog.fraggod.net/2019/01/10/tui-console-dechexbinary-converter-tool.html

##### [maildir-cat](maildir-cat)

Python script to iterate over all messages in all folders of a maildir and
print (decoded) headers and plain + html body of each (decoded) message, with
every line prefixed by its filename.

Intended use is to produce a text dump of a maildir for searching or processing
it via any simple tools like grep or awk.

So using e.g. `maildir-cat | grep 'important-word'` will produce same output
as `grep -r 'important-word' email-texts/` would if emails+headers were dumped
as simple text files there.

Can also be pointed to maildir subdirs (same thing) or individual files.\
Uses python stdlib email.* modules for all processing.

##### [dns-update-proxy](dns-update-proxy)

Small py3/asyncio UDP listener that receives ~100B `pk || box(name:addr)`
libnacl-encrypted packets, decrypts (name, addr) tuples from there,
checking that:

- Public key of the sender is in `-a/--auth-key` list.
- Name doesn't resolve to same IP already, among any others (`-c/--check` option).
- Name has one of the allowed domain suffixes (`-d/--update` option).

If all these pass, specified BIND-format zone-file (for e.g. [nsd]) is updated,
or DNS service API used to same effect, with several retries on any fails
(`-r/--retry` option) and rate-limiting, as well as `--debug` logging.

Useful wrapper for auto-updating names in delegated nsd-managed zone,
or doing same via DNS APIs that only provide all-or-nothing access,
while you want to setup convenience names from some shared-access VM,
without giving away creds for the whole account on these services,
with all other names and subdomains there.

Example snippet for sending update packets:

``` python
import socket, time, libnacl.public, base64, pathlib as pl

b64_decode = lambda s: ( base64.urlsafe_b64decode
if '-' in s or '_' in s else base64.standard_b64decode )(s)

class Conf:
proxy_addr = 'dns-proxy.host.net'
proxy_pk = 'wnQvfuzUNyjDgFhPa23y0z5iXJl8TuZ+rdL0G3vefxQ='
sk_file = 'local_key.secret' # use e.g. "wg genkey" or libnacl
key = libnacl.public.SecretKey(b64_decode(pl.Path(sk_file).read_text()))
box = libnacl.public.Box(key, b64_decode(proxy_pk))
encrypt = lambda s, msg: s.key.pk + s.box.encrypt(msg)
proxy_conf = Conf()

def update_dns(conf, name, addr):
msg = conf.encrypt(f'{name}:{addr}'.encode())
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
for delay in [0.1, 0.5, 1, 3, 0]:
try: s.sendto(msg, conf.proxy_addr)
except (socket.gaierror, socket.error): pass
if delay: time.sleep(delay)

update_dns(proxy_conf, 'my.ddns.host.net', '1.2.3.4')
```

[nsd]: https://wiki.alpinelinux.org/wiki/Setting_up_nsd_DNS_server

##### [dns-test-daemon](dns-test-daemon)

Python + [async_dns] authoritative DNS resolver daemon to
return hashed-name results for testing DNS resolver operation.

For example:

``` console
% ./dns-test-daemon -k hash-key -b 127.0.0.1:5533 &
% dig -p5533 @127.0.0.1 aaaa test.com
...
test.com. 300 IN AAAA eb5:7823:f2d2:2ed2:ba27:dd79:a33e:f762
...
```

Here, for AAAA "test.com" query, script returned first 16 bytes of
"blake2s(test.com, key=hash-key, person=dnstd.1)" hash digest
as a reponse (converted to address via inet_ntop).

Its purpose is to be run as an authoritative resolver for some stub zone
forwarded to it, e.g. "\*.test.mydomain.com", and then be able to make sure that
any local DNS resolver works by querying e.g. "12345.test.mydomain.com" and
checking that resulting address hash matches expected value (dependent only on
queried name, hash key and that hardcoded person= string).

To run script in tester-client mode, simply pass it a name to test, along with
same `-k/--hash-key` parameter as for daemon on the other end, e.g.:

% ./dns-test-daemon -k hash-key random-stuff.test.mydomain.com
% ./dns-test-daemon -k hash-key --debug @.test.mydomain.com

It will exit with non-zero code if result is missing or doesn't match expected
value in any way.

Does not import/use or require asyncio and async_dns modules in client mode.

Its `-c/--continuous` mode can be used together with systemd to kick/restart
unreliable resolver daemon (e.g. unbound) when it hangs or fails in other ways:

``` ini
[Service]
Type=exec
User=dnstd
ExecStart=dns-test-daemon -c 150:6:100 -p 1.1.1.1 @.test.mydomain.com
ExecStopPost=+bash -c '[[ "$$SERVICE_RESULT" = success ]] || systemctl try-restart unbound'

# Using RestartForceExitStatus=53 should prevent unbound restarts on script bugs
RestartForceExitStatus=53
RestartSec=5min

[Install]
WantedBy=multi-user.service
```

Note `-p 1.1.1.1` ping-option there to avoid restarting the daemon if whole
network is down, which runs "fping" to check that on detected DNS failures.

[async_dns]: https://github.com/gera2ld/async_dns

##### [nginx-access-log-stat-block](nginx-access-log-stat-block)

Python/ctypes script to be used alongside [nginx-stat-check] module, reliably
tailing any kind of access.log-like file(s) where first (space-separated) field
is IP address and creating files with name corresponding to these in specified db_dir.

nginx-stat-check module then allows to use `stat_check /some/db_dir/$remote_addr;`
in nginx.conf to return 403 for all addresses processed in this way.

Created files are automatically renamed and cleaned-up after specified
unblock/forget-timeouts and block-timeout either get extended or multiplied by
specified k value (2x default) on repeated blocks after expiry.

Intended use it to block stupid bots and whatever spammers that don't care about
robots.txt when these access some honeypot-file on nginx level (with proper 403
on specific URL paths), which normally should never be requested.

I.e. bots that are stupidly re-indexing giant file dumps or whatever dynamic
content every N minutes.

Example nginx.conf snippet:

load_module /usr/lib/nginx/modules/ngx_http_stat_check.so;
log_format stat-block '$remote_addr :: $time_iso8601 "$http_referer" "$http_user_agent"';
...

location = /distro/package/mirror/open-and-get-banned.txt {
alias /srv/pkg-mirror/open-and-get-banned.txt;
access_log /var/log/nginx/bots.log stat-block;
}

location /distro/package/mirror {
alias /srv/pkg-mirror;
autoindex on;
stat_check /tmp/stat-block/$remote_addr;
}

And run script to populate `/tmp/stat-block/` path from bots.log:

% ./nginx-access-log-stat-block --debug /tmp/stat-block/ /var/log/nginx/bots.log

Check `-h/--help` output for default block-timeout and such values.

Uses inotify to tail files via ctypes, detects log rotation but NOT truncation
(use with append/remove-only logs), can tail multiple wildcard-matching files
in a directory, closes opened/tailed logs after timeout.

Always opens files at the end, so can loose a line or two due to that,
which is fine for intended purpose (bots spam requests anyway).

[nginx-stat-check]: https://github.com/mk-fg/nginx-stat-check

##### [sys-wait](sys-wait)

Bash script to check and wait for various system conditions,
files, processes or thresholds like load average or PSI values.

Random examples:

% sys-wait -l 3 && run-less-heavy-task
% sys-wait --load15 5 && run-next-heavy-task
% sys-wait -f /some/file/appeared && process-file
% sys-wait -F /file/to-be-removed && run-stuff

Helps to avoid writing those annoyingly-common
`while :; do some-check || break; sleep 60; done; run-other-stuff`
when something heavy/long is already running and you just don't
have the heart to break and reschedule it properly.

Mostly used to need for pgrep in a loop, but these days util-linux includes
pidwait binary, which does the job without this wrapper.

##### [yt-feed-to-email](yt-feed-to-email)

Python + [feedparser] RSS-to-email notification script for YouTube RSS feeds.

Can process OPML of current YT subscriptions (from
)
or work with one-per-line list of channel/video RSS feed links.

Remembers last feed state(s) via auto-rotating log, uses [EWMA]
to calculate delay between checks based on feed update interval.

Useful to keep track of YT channel updates via read/unread status in some
dedicated mailbox folder, and click-open video links from there in mpv,
like one could before Aug 2020 when google decided to stop sending all update
notification emails on that platform.

[feedparser]: https://pythonhosted.org/feedparser/
[EWMA]: https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average

##### [color-b64sort](color-b64sort)

Tool to filter, sort and compress list of colors - aka color palette - into
base64, to then use as a compact blob in visualization scripts easily.

- Input: a list of hex-encoded colors, separated by any spaces/newlines.

- Filtering:

Removes colors too close to specified background color
(using specified Delta E CIE 2000 color-diff threshold).

Compares colors all-to-all, and removes ones that are too close to each other,
with a similar configurable threshold.

- Ordering:

Picks next color based on min(deltas-with-others) value, to get the most
distinct color on every step.

This is further configured by using higher weights of min(deltas-with-n-last)
colors, so that next pick ends up being as distinct as possible from N ones
that are right before it first, and then the rest of them.

Current default for `-k/--sort-delta-keys` "weight:count" list is "0.3:5 0.2:10
0.1:20", with leftover 0.4 weight used for min(deltas-with-all-picked) value.

- Output:

Urlsafe-base64 of concatenated 3-byte color values in RGB order,
instead of more bulky "lines of hex-encoded colors" or other color-spec types,
to hardcode without taking too much space.

Intended use it to have output color list of 50+ values, and then pick them in
order (for chart lines, tree branches, table row/cell backgrounds, etc), which
should return most distinctive colors first, without resorting to repetition as
quickly as with e.g. D3.js fixed 10/20-color palettes.

There are many great tools like ["i want hue"] that can be used to generate input
color list for this script, with features like accounting for color blindness types,
but it can be just a sequence of points from any nice gradient too - input
ordering or similarity should not matter.

It's a small python script, which uses [colormath] module for Delta E CIE 2000
color-diff calculations.
Can take some time to run with long lists due to how all\*all combinatorics work,
but using pypy instead of cpython can speed that up a lot.

["i want hue"]: https://medialab.github.io/iwanthue/
[colormath]: https://python-colormath.readthedocs.io/

##### [svg-tweak](svg-tweak)

Small python script to change SVG files, according to specified options.

For example, if an image viewer displays transparent SVG with back text on a black
background (as one solid-black rectangle), `svg-tweak -b '#fff' file.svg` can fix it.

SVGs are XML text, so aren't difficult to change like that, but old unix cli tools
like sed and awk aren't great for that, and tend to require a bunch of extra logic.

##### [unix-socket-links](unix-socket-links)

Python wrapper around `ss -xp` output, processing disjointed unix socket
connection table (with pids on only one end of those), into more readable
aggregated ` :: :: ` list.

`ss -xp src ` is closest to this functionality, but doesn't actually list
clients connected there, e.g. for X11 socket it lists same Xorg process uselessly
for each connection, instead of actual X apps connected to that socket.

Use-case is to quickly check what's connected to some socket path
(which maybe you don't remember exactly), by printing a short list of all
of them with listener/client pids, when some connection hangs or ssh-agent
asks for fido2 touch-check unexpectedly.

Has more human-readable `-p/--pretty` mode and more traditional disaggregated
`-c/--conns` mode for listing specific connections instead of just processes.

See ["List connected processes for unix sockets" blog post] for some usage examples.

["List connected processes for unix sockets" blog post]:
https://blog.fraggod.net/2024/08/06/list-connected-processes-for-unix-sockets-on-linux.html

##### [tcpdump-translate](tcpdump-translate)

Wrapper script for running `tcpdump -ln` (unbuffered lines, no dns), to translate,
color-highlight and optionally filter-by specified addresses and network prefixes.

There are couple images showing what it does in ["Adding color to tcpdump" blog post].

["Adding color to tcpdump" blog post]:
https://blog.fraggod.net/2024/09/30/adding-color-to-tcpdump-makes-a-ton-of-difference.html

Intended use is to match known hosts or networks in the output, while leaving
all other addresses intact, without going to DNS PTR records or anything like that.

For example, with the following `ipv6-debug.tt` file:
```
# " [!]" specs, newline/comma separated
# Exact-match full address should end with "/". Example: 1.2 mynet, 1.2.3.4/ myaddr

2a01:4f8:c27:34c2: A.net:
2a01:4f8:c27:34c2::2/ [A]

2a01:4f8:c27:34c2:8341:8768:e26:83ff/ [A.ns] !red

2a02:13d1:22:6a0 B.net
2a02:13d1:22:6a01::1/ [B]

2a02:13d1:22:6a00:2a10:6f67:8c0:60ae/ [B.host-X] !bold-green
2a02:13d1:22:6a00:de8a:12c8:e85:235f/ [B.laptop] !bold-bright-yellow

127.0.0. lo4., :: lo6.
```

And then running e.g. `tcpdump -i eth0 | ./tcpdump-translate -m ipv6-debug.tt`
will produce translated output (also truncated to terminal width by default):
```
11:40:00.641680 IP6 A.net:8341:865e:e26:8401.31788 > [B.laptop].31788: UDP, length 32
11:41:49.868243 IP6 [A.ns].31788 > B.net0:de8c::28f1.31788: UDP, length 148
11:41:51.148385 IP6 [A.ns].31788 > B.net0:de8c::28f2.31788: UDP, length 148
...
11:42:23.735140 IP6 [A.ns].31788 > [B.laptop].31788: UDP, length 148
11:42:24.801590 IP6 [A.ns].31788 > [B].11446: UDP, length 148
11:42:26.286887 IP6 [B.host-X].31788 > [A.ns].31788: UDP, length 32
11:42:26.287739 IP6 [B.host-X].31788 > [A.ns].31788: UDP, length 148
11:42:26.288301 IP6 [A.ns].31788 > [B.host-X].31788: UDP, length 92
11:42:26.350673 IP6 [B.host-X].31788 > [A.ns].31788: UDP, length 32
11:42:29.068373 IP6 [A.ns].31788 > [B.laptop].31788: UDP, length 148
11:42:29.573134 IP6 [A.ns].47504 > [B].80: Flags [S], seq 3249847667, win 33120,
11:42:29.638883 IP6 [B].80 > [A.ns].47504: Flags [S.], seq 271826300, ack 324984
11:42:29.639081 IP6 [A.ns].47504 > [B].80: Flags [.], ack 1, win 259, options
...
11:42:29.705541 IP6 [A.ns].47504 > [B].80: Flags [F.], seq 75, ack 375, win 257,
11:42:29.770506 IP6 [B].80 > [A.ns].47504: Flags [F.], seq 375, ack 76, win 251,
11:42:29.770583 IP6 [A.ns].47504 > [B].80: Flags [.], ack 376, win 257, options
11:42:29.921720 IP6 [A.ns].31788 > [B].11446: UDP, length 148
```

Where replacements are done either for full addresses or their string prefixes
(not CIDR prefixes, simple string match-replace).

Without this, IPv6es in output above

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/mk-fg/fgtk

Awesome Lists containing this project

README