{"id":13586104,"url":"https://github.com/mk-fg/fgtk","last_synced_at":"2025-04-04T18:08:28.709Z","repository":{"id":2829006,"uuid":"3831498","full_name":"mk-fg/fgtk","owner":"mk-fg","description":"A set of a misc tools to work with files and processes","archived":false,"fork":false,"pushed_at":"2024-10-29T19:03:52.000Z","size":3621,"stargazers_count":157,"open_issues_count":0,"forks_count":34,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-10-29T19:04:19.362Z","etag":null,"topics":["bash","collection","dev","distro","git","linux","metrics","misc","python","scraps","sysadmin","systemd","tools","utilities"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mk-fg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":"audit-follow","citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-03-26T09:58:03.000Z","updated_at":"2024-10-29T19:03:56.000Z","dependencies_parsed_at":"2023-07-05T20:03:07.209Z","dependency_job_id":"dec166cd-af8b-4b8d-a1ab-999e1e4d4e14","html_url":"https://github.com/mk-fg/fgtk","commit_stats":{"total_commits":1442,"total_committers":3,"mean_commits":480.6666666666667,"dds":0.001386962552011095,"last_synced_commit":"07351d8c21c1617c4e2a76a6cb2b29340fe6d31d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Ffgtk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Ffgtk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Ffgtk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Ffgtk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mk-fg","download_url":"https://codeload.github.com/mk-fg/fgtk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247226215,"owners_count":20904465,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bash","collection","dev","distro","git","linux","metrics","misc","python","scraps","sysadmin","systemd","tools","utilities"],"created_at":"2024-08-01T15:05:19.949Z","updated_at":"2025-04-04T18:08:28.687Z","avatar_url":"https://github.com/mk-fg.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# fgtk\n\nA set of a misc tools to work with files and processes.\n\nVarious oldish helper scripts/binaries I wrote to help myself with\nday-to-day tasks.\n\nLicense for all scripts is WTFPL (public domain-ish - [see below](#hdr-license__wtfpl_)),\nfeel free to just copy and use these in whatever way you like.\n\nRepository URLs:\n\n - \u003chttps://github.com/mk-fg/fgtk\u003e\n - \u003chttps://codeberg.org/mk-fg/fgtk\u003e\n - \u003chttps://fraggod.net/code/git/fgtk\u003e\n\nContents - links to doc section for each script here:\n\n- [\\[-root-\\] Various CLI/system things](#hdr--root-___various_cli_system_things)\n\n    - [File/dir/fs management](#hdr-file_dir_fs_management)\n\n        - [scim](#hdr-scim)\n        - [run_cmd_pipe.nim](#hdr-run_cmd_pipe.nim)\n        - [findx](#hdr-findx)\n        - [patch-nspawn-ids](#hdr-patch-nspawn-ids)\n        - [bindfs-idmap](#hdr-bindfs-idmap)\n        - [docker-ln](#hdr-docker-ln)\n        - [fast-disk-wipe](#hdr-fast-disk-wipe)\n        - [lsx](#hdr-lsx)\n        - [trunc-filenames](#hdr-trunc-filenames)\n\n    - [Various file-data processing tools](#hdr-various_file-data_processing_tools)\n\n        - [repr](#hdr-repr)\n        - [color](#hdr-color)\n        - [resolve-hostnames](#hdr-resolve-hostnames)\n        - [resolve-conf](#hdr-resolve-conf)\n        - [temp-patch](#hdr-temp-patch)\n        - [term-pipe](#hdr-term-pipe)\n        - [yaml-to-pretty-json](#hdr-yaml-to-pretty-json)\n        - [yaml-flatten](#hdr-yaml-flatten)\n        - [yaml-diff](#hdr-yaml-diff)\n        - [hz](#hdr-hz)\n        - [liac](#hdr-liac)\n        - [html-embed](#hdr-html-embed)\n        - [someml-indent](#hdr-someml-indent)\n        - [hashname](#hdr-hashname)\n        - [hhash](#hdr-hhash)\n        - [crypt](#hdr-crypt)\n\n    - [Kernel sources/build/version management](#hdr-kernel_sources_build_version_management)\n\n        - [kernel-patch](#hdr-kernel-patch)\n        - [kernel-conf-check](#hdr-kernel-conf-check)\n        - [clean-boot](#hdr-clean-boot)\n\n    - [ZNC log helpers](#hdr-znc_log_helpers)\n\n        - [znc-log-aggregator](#hdr-znc-log-aggregator)\n        - [znc-log-reader](#hdr-znc-log-reader)\n\n    - [systemd-related](#hdr-systemd-related)\n\n        - [systemd-dashboard](#hdr-systemd-dashboard)\n        - [systemd-watchdog](#hdr-systemd-watchdog)\n        - [cgrc](#hdr-cgrc)\n\n    - [SSH and WireGuard related](#hdr-ssh_and_wireguard_related)\n\n        - [ssh-fingerprint](#hdr-ssh-fingerprint)\n        - [ssh-keyparse](#hdr-ssh-keyparse)\n        - [ssh-key-init](#hdr-ssh-key-init)\n        - [ssh-tunnel](#hdr-ssh-tunnel)\n        - [ssh-reverse-mux-server / ssh-reverse-mux-client](#hdr-ssh-reverse-mux-server_ssh-reverse-mux-client)\n        - [wg-mux-server / wg-mux-client](#hdr-wg-mux-server___wg-mux-client)\n        - [ssh-tunnels-cleanup](#hdr-ssh-tunnels-cleanup)\n        - [mosh-nat / mosh-nat-bind.c](#hdr-mosh-nat___mosh-nat-bind.c)\n        - [tping](#hdr-tping)\n\n    - [WiFi / Bluetooth helpers](#hdr-wifi___bluetooth_helpers)\n\n        - [adhocapd](#hdr-adhocapd)\n        - [wpa-systemd-wrapper](#hdr-wpa-systemd-wrapper)\n        - [timed-ble-beacon](#hdr-timed-ble-beacon)\n        - [timed-ble-beacon-mpy-led](#hdr-timed-ble-beacon-mpy-led)\n\n    - [Misc](#hdr-misc)\n\n        - [at](#hdr-at)\n        - [sleepc](#hdr-sleepc)\n        - [wgets](#hdr-wgets)\n        - [mail](#hdr-mail)\n        - [passgen](#hdr-passgen)\n        - [urlparse](#hdr-urlparse)\n        - [ip-ext](#hdr-ip-ext)\n        - [blinky](#hdr-blinky)\n        - [openssl-fingerprint](#hdr-openssl-fingerprint)\n        - [nsh](#hdr-nsh)\n        - [pam-run](#hdr-pam-run)\n        - [primes](#hdr-primes)\n        - [boot-patcher](#hdr-boot-patcher)\n        - [audit-follow](#hdr-audit-follow)\n        - [tui-binary-conv](#hdr-tui-binary-conv)\n        - [maildir-cat](#hdr-maildir-cat)\n        - [dns-update-proxy](#hdr-dns-update-proxy)\n        - [dns-test-daemon](#hdr-dns-test-daemon)\n        - [nginx-access-log-stat-block](#hdr-nginx-access-log-stat-block)\n        - [sys-wait](#hdr-sys-wait)\n        - [yt-feed-to-email](#hdr-yt-feed-to-email)\n        - [color-b64sort](#hdr-color-b64sort)\n        - [svg-tweak](#hdr-svg-tweak)\n        - [unix-socket-links](#hdr-unix-socket-links)\n        - [tcpdump-translate](#hdr-tcpdump-translate)\n\n- [\\[dev\\] Dev tools](#hdr-dev___dev_tools)\n\n    - [indent-replace](#hdr-indent-replace)\n    - [indent-braces](#hdr-indent-braces)\n    - [golang_filter](#hdr-golang_filter)\n    - [distribute_regen](#hdr-distribute_regen)\n    - [darcs_bundle_to_diff](#hdr-darcs_bundle_to_diff)\n    - [git-nym](#hdr-git-nym)\n    - [git-meld](#hdr-git-meld)\n    - [catn](#hdr-catn)\n    - [git_terminate](#hdr-git_terminate)\n    - [git_contains](#hdr-git_contains)\n    - [gtk-val-slider](#hdr-gtk-val-slider)\n    - [git-version-bump-filter](#hdr-git-version-bump-filter)\n    - [git-prepare-commit-msg-hook](#hdr-git-prepare-commit-msg-hook)\n    - [markdown-checks](#hdr-markdown-checks)\n\n- [\\[backup\\] Backup helpers](#hdr-backup___backup_helpers)\n\n    - [ssh-r-sync / ssh-r-sync-recv](#hdr-ssh-r-sync___ssh-r-sync-recv)\n    - [ssh-dump](#hdr-ssh-dump)\n    - [zfs-snapper](#hdr-zfs-snapper)\n    - [btrfs-snapper](#hdr-btrfs-snapper)\n    - [dir-snapper](#hdr-dir-snapper)\n\n- [\\[hsm\\] FIDO2 / PIV / etc smartcard stuff](#hdr-hsm___fido2___piv___etc_smartcard_stuff)\n\n    - [fido2-hmac-desalinate.c](#hdr-fido2-hmac-desalinate.c)\n    - [fido2_hmac_boot.nim](#hdr-fido2_hmac_boot.nim)\n    - [secret-token-backup](#hdr-secret-token-backup)\n\n- [\\[desktop\\] Linux desktop stuff](#hdr-desktop___linux_desktop_stuff)\n\n    - [\\[desktop/uri_handlers\\]](#hdr-desktop_uri_handlers__)\n\n    - [\\[desktop/media\\]](#hdr-desktop_media__)\n\n        - [toogg](#hdr-toogg)\n        - [tomkv](#hdr-tomkv)\n        - [totty](#hdr-totty)\n        - [split](#hdr-split)\n        - [audio-split-m4b](#hdr-audio-split-m4b)\n        - [audio-split-flac-cue](#hdr-audio-split-flac-cue)\n        - [video-concat-xfade](#hdr-video-concat-xfade)\n        - [pick-tracks](#hdr-pick-tracks)\n        - [twitch_vod_fetch](#hdr-twitch_vod_fetch)\n        - [ytdl-chan](#hdr-ytdl-chan)\n        - [streamdump](#hdr-streamdump)\n        - [image-compact](#hdr-image-compact)\n\n    - [\\[desktop/notifications\\]](#hdr-desktop_notifications__)\n\n        - [exec](#hdr-exec)\n        - [power](#hdr-power)\n        - [logtail](#hdr-logtail)\n        - [dovecot-mail](#hdr-dovecot-mail)\n        - [icon](#hdr-icon)\n        - [aqm-alerts](#hdr-aqm-alerts)\n        - [dev-nodes](#hdr-dev-nodes)\n\n    - [\\[desktop\\] others](#hdr-desktop___others)\n\n        - [vfat_shuffler](#hdr-vfat_shuffler)\n        - [fan_control](#hdr-fan_control)\n        - [emms-beets-enqueue](#hdr-emms-beets-enqueue)\n        - [ff_backup](#hdr-ff_backup)\n        - [ff-cli](#hdr-ff-cli)\n        - [bt_agent](#hdr-bt_agent)\n        - [alarm](#hdr-alarm)\n        - [acpi-wakeup-config](#hdr-acpi-wakeup-config)\n        - [olaat](#hdr-olaat)\n        - [blinds](#hdr-blinds)\n        - [evdev-to-xev](#hdr-evdev-to-xev)\n        - [exclip](#hdr-exclip)\n        - [xdpms](#hdr-xdpms)\n        - [xiwait](#hdr-xiwait)\n        - [xkbledq](#hdr-xkbledq)\n        - [rss-get](#hdr-rss-get)\n        - [qr](#hdr-qr)\n        - [gtk-color-calc](#hdr-gtk-color-calc)\n        - [filetag](#hdr-filetag)\n        - [hamster-tally](#hdr-hamster-tally)\n        - [feh-screen](#hdr-feh-screen)\n\n- [\\[vm\\] VM scripts](#hdr-vm___vm_scripts)\n\n- [\\[bpf\\] Linux eBPF filters](#hdr-bpf___linux_ebpf_filters)\n\n- [\\[arch\\] ArchLinux(+ARM)](#hdr-arch___archlinux__arm_)\n\n    - [elf-deps](#hdr-elf-deps)\n    - [pacman-manifest](#hdr-pacman-manifest)\n    - [pacman-extra-files](#hdr-pacman-extra-files)\n    - [pacman-pacnew](#hdr-pacman-pacnew)\n    - [pacman-fsck](#hdr-pacman-fsck)\n    - [tar-strap](#hdr-tar-strap)\n    - [can-strap](#hdr-can-strap)\n\n- [\\[alpine\\] Alpine Linux](#hdr-alpine___alpine_linux)\n\n    - [manifest](#hdr-manifest)\n\n- [\\[metrics\\] Charts and metrics](#hdr-metrics___charts_and_metrics)\n\n    - [rrd-sensors-logger](#hdr-rrd-sensors-logger)\n    - [graphite-scratchpad](#hdr-graphite-scratchpad)\n    - [gnuplot-free](#hdr-gnuplot-free)\n    - [d3-line-chart-boilerplate](#hdr-d3-line-chart-boilerplate)\n    - [d3-histogram-boilerplate](#hdr-d3-histogram-boilerplate)\n    - [d3-temp-rh-sensor-tsv-series-chart](#hdr-d3-temp-rh-sensor-tsv-series-chart)\n    - [d3-du-disk-space-usage-layout](#hdr-d3-du-disk-space-usage-layout)\n    - [prometheus-snmp-iface-counters-exporter](#hdr-prometheus-snmp-iface-counters-exporter)\n    - [prometheus-grafana-simplejson-aggregator](#hdr-prometheus-grafana-simplejson-aggregator)\n    - [systemd-cglog](#hdr-systemd-cglog)\n    - [load-check-logger](#hdr-load-check-logger)\n\n- [\\[cron-checks\\] Diff/alert checks for crontab](#hdr-cron-checks_things_to_run_from_crontab_s.0UQD)\n\n    - [df](#hdr-df)\n    - [attrs](#hdr-attrs)\n    - [git-manifest](#hdr-git-manifest)\n    - [systemd](#hdr-systemd)\n\n- [\\[scraps\\]](#hdr-scraps__)\n\n    - [rsync-diff](#hdr-rsync-diff)\n    - [pcap-process](#hdr-pcap-process)\n    - [log-tail-check](#hdr-log-tail-check)\n    - [resize-rpi-fat32-for-card](#hdr-resize-rpi-fat32-for-card)\n    - [asciitree-parse](#hdr-asciitree-parse)\n    - [glusterfs-xattr-trusted-to-user](#hdr-glusterfs-xattr-trusted-to-user)\n    - [led-blink-arg](#hdr-led-blink-arg)\n    - [led-blink-seq](#hdr-led-blink-seq)\n    - [gue-tunnel](#hdr-gue-tunnel)\n    - [wifi-client-match](#hdr-wifi-client-match)\n    - [mem-search-replace](#hdr-mem-search-replace)\n    - [gpm-track](#hdr-gpm-track)\n    - [rsyslogs](#hdr-rsyslogs)\n    - [relp-test](#hdr-relp-test)\n    - [exec.c](#hdr-exec.c)\n    - [sqlite-python-concurrency-test](#hdr-sqlite-python-concurrency-test)\n    - [numfmt.awk](#hdr-numfmt.awk)\n    - [nft-ddos](#hdr-nft-ddos)\n\n\n\n\u003ca name=hdrx-scripts\u003e\u003c/a\u003e\n## Scripts\n\n\n\n\u003ca name=hdr--root-___various_cli_system_things\u003e\u003c/a\u003e\n### \\[-root-\\] Various CLI/system things\n\n\n\n\u003ca name=hdr-file_dir_fs_management\u003e\u003c/a\u003e\n#### File/dir/fs management\n\nFile/link/dir and filesystem structure manipulation tools.\n\n\u003ca name=hdr-scim\u003e\u003c/a\u003e\n##### [scim](scim)\n\nNon-interactive CLI tool to keep a list of files to symlink or copy into/from\nsome \"dotfiles\" configuration dir or repository, and keep/check/update/restore\nmetadata manifest for these files.\n\nKeeps track of ACLs, POSIX capabilities and xattrs for metadata, runs file\ndiffs for file copies and links, supports a bunch of neat symlinking options\n(like using relative symlinks, relative symlinks into symlinked repo-dir, etc).\n\nIdea is to keep links and metadata manifest files in some configuration repo,\nand run the tool occasionally after system updates or manual changes to pull\nupdated files into repo, update files on fs from the repo, fix links/permissions\non fs, copy/add new ones, etc - all manifest/maintenance ops done via this script.\n\nFormat for links-list looks something like this:\n\n    .gitconfig -\u003e .git/config\n    /usr/share/zoneinfo/Asia/Yekaterinburg -\u003e /etc/localtime\n    bpf -\u003e /etc/bpf\n    zshrc \u003e /etc/zsh/zshrc\n    kernel-config \u003e /usr/src/linux/.config\n    myapp/secret.conf -\u003e /etc/myapp/secret.conf\n    myapp/suid.bin -\u003e /usr/local/bin/myapp\n    myapp/caps.bin -\u003e /usr/local/bin/myapp-helper\n\nAnd metadata is also a simple plaintext file, with fancier stuff towards the\nend of lines, on paths where it's used/needed:\n\n    .gitconfig root:root:644\n    bpf root:wheel:750\n    zshrc root:root:644\n    kernel-config root:wheel:664\n    myapp/secret.conf root:root:600\n    myapp/suid.bin root:root:4711\n    myapp/caps.bin root:root:4700/EP:net_raw/u::rwx,u:netuser:--x,g::r-x,m::r-x,o::---\n\nIn addition to lists, there're separate links/meta exclude-files with regexps of\npaths to not warn about being missing in links-list or track metadata for.\n\nOnly needs python3 to run, has bundled implementation for parsing/encoding\nmodern linux ACLs/capabilities extended attributes.\nUses `git diff --no-index` for `--diff-cmd` by default, as it is very fast,\nhas nice colors and should be widely available.\n\nStarted as a [cfgit project] long time ago, evolved away into this more generic\n(and not necessarily git-related) tool.\n\n[cfgit project]: https://fraggod.net/code/git/configit/\n\n\u003ca name=hdr-run_cmd_pipe.nim\u003e\u003c/a\u003e\n##### [run_cmd_pipe.nim](run_cmd_pipe.nim)\n\nSmall tool to match lines from stdin according to ini config file\nand run commands for any matching regexps specified there.\nIntended as a long-running handler for monitoring some process' output,\ne.g. monitor some log via `tail -F file.log`, or react to [fanotify]\nfilesystem updates from [fatrace] efficiently.\n\nFor example, with `myapp-changes.conf` file like this:\n\n``` ini\n# Add 10s delay for changes to settle before running commands\ndelay = 10_000\n\n[data-file-updates]\nregexp = : \\S*[WD+\u003c\u003e]\\S* */srv/myapp/data-files(/[^/]+)?$\nrun = myapp process-new-data /srv/myapp/data-files\n# regexp-env-var = RCP_MATCH -- \"run\" command will get this in env by default\n# regexp-env-group = 1 -- regexp group to put into regexp-env-var, 0 - full match\n# regexp-run-group = 1 -- to run/delay/cooldown commands based on matched group\n\n[config-updates]\nregexp = : \\S*[WD+\u003c\u003e]\\S* */srv/myapp/config(/.*)?$\nrun = pkill -x HUP myapp\n```\n\n...tool can be run as `fatrace | run_cmd_pipe myapp-changes.conf` (or exec\ninput-command without shell via `... -- cmd args...` by itself), to process\nany file-change events and run relevant commands to react to those in a daemon loop.\n\nCan have cooldown and debouncing delay for rules, reloads config-file on SIGHUP,\nruns only one process per rule at a time, has small mem footprint, no deps, etc etc.\n`-h/--help` output has more info on configuration format and cli opts.\n\nBuild with:\n`nim c -d:release --opt:size run_cmd_pipe.nim \u0026\u0026 strip run_cmd_pipe`\n\nOne interesting use I've found in combination with [fatrace] is to\n[monitor and synchronize local containers, as well as handle events from those].\n\n[fatrace]: https://github.com/martinpitt/fatrace\n[fanotify]: https://lwn.net/Articles/339253/\n[monitor and synchronize local containers, as well as handle events from those]:\n  https://blog.fraggod.net/2024/01/09/ab-using-fanotify-as-a-container-eventmessage-bus.html\n\n\u003ca name=hdr-findx\u003e\u003c/a\u003e\n##### [findx](findx)\n\nWrapper around GNU find (from [findutils]) to accept paths at the end of argv\nif none are passed before query.\n\nMakes it somewhat more consistent with most other commands that accept\noptions and a lists of paths (almost always after opts),\nbut still warns when/if reordering takes place.\n\nNo matter how many years I'm using that tool, still sometimes type paths\nafter query there, so decided to patch around that frustrating issue one day.\n\n[findutils]: https://www.gnu.org/software/findutils/\n\n\u003ca name=hdr-patch-nspawn-ids\u003e\u003c/a\u003e\n##### [patch-nspawn-ids](patch-nspawn-ids)\n\nPython script to \"shift\" or \"patch\" uid/gid values with new container-id\naccording to systemd-nspawn schema, i.e. set upper 16-bit to specified\ncontainer-id value and keep lower 16 bits to uid/gid inside the container.\n\nSimilar operation to what systemd-nspawn's --private-users-chown option does\n(described in nspawn-patch-uid.c), but standalone, doesn't bother with ACLs or\nchecks on filesystem boundaries.\n\nMain purpose is to update uids when migrating systemd-nspawn containers or\nadding paths/filesystems to these without clobbering ownership info there.\n\nShould be safe to use anywhere, as in most non-nspawn cases upper bits of\nuid/gid are always zero, hence any changes can be easily reverted by running\nthis tool again with -c0.\n\n\u003ca name=hdr-bindfs-idmap\u003e\u003c/a\u003e\n##### [bindfs-idmap](bindfs-idmap)\n\n[bindfs] wrapper script to setup id-mapping from uid of the mountpoint\nto uid/gid of the source directory.\n\nI.e. after `bindfs-idmap /var/lib/machines/home/src-user ~dst-user/tmp`,\n`~dst-user/tmp` will be accessible to dst-user as if they were src-user,\nwith all operations proxied to src-user's dir.\n\nAnything created under `~dst-user/tmp` will have uid/gid of the src dir.\n\nUseful to allow temporary access to some uid's files in a local container\nto a user id in a main namespace.\n\nFor long-term access (e.g. for some daemon), there probably are better options\nthan such bindfs hack - e.g. bind/idmapped mounts, shared uids/gids, ACLs, etc.\n\n[bindfs]: https://bindfs.org/\n\n\u003ca name=hdr-docker-ln\u003e\u003c/a\u003e\n##### [docker-ln](docker-ln)\n\nSimple bash script to symlink uppermost \"merged\" overlayfs layer of a running\ndocker-compose setup container, to allow easy access to temporary files there.\n\nUseful for testing stuff without the need to rebuild and restart whole container\nor a bunch of compose stuff after every one-liner tweak to some script that's\nsupposed to be running in there, or to experiment-with and debug things.\n\nThese paths are very likely to change between container and docker-compose\nrestarts for many reasons, so such symlinks are generally only valid during\ncontainer runtime, and script needs a re-run to update these too.\n\n\u003ca name=hdr-fast-disk-wipe\u003e\u003c/a\u003e\n##### [fast-disk-wipe](fast-disk-wipe.c)\n\nVery simple \"write 512B, skip N * 512B, repeat\" binary for wiping some block\ndevice in a hurry.\n\nIdea is not to erase every trace of data or to hide it, but just to make files\nprobabilistically unusable due to such junk blocks all over the place.\n\nWith low-enough intervals it should also corrupt filesystem pretty badly,\nmaking metadata hard to access.\n\nFast loop of 512B writes to a device directly will likely hang that binary until\nit's done, as that's how such direct I/O seem to work on linux.\n\nWrites only stop when write() or lseek() starts returning errors, so using this\non some extendable file will result in it eating up all space available to it.\n\nSee head of the file for build and usage info.\n\n\u003ca name=hdr-lsx\u003e\u003c/a\u003e\n##### [lsx](lsx)\n\nMore functionality similar to common \"ls\" tool, to list files in some specific\nways that are occasionally useful. All those are available via various options -\nsee `-h/--help` for a full list.\n\nFor example, to print up to N `-a/--adjacent` files (within some specific ordering):\n``` console\n% lsx -aS data/chunk-12345.bin  # default up to 10 before/after, w/ S=size ordering\n% lsx -a 50as data/chunk-13.bin # only 50 files larger than specified one\n% lsx -a 5bt myapp/state.log    # up to 5 logs right before state.log by mtime\n% lsx -fa a3 logs/20230515.log  # 3 log-files (-f/--files) with names after that one\n```\n\nOr files within `-t/--mtime` vicinity/ranges:\n``` console\n% lsx -t 1h cache/a/bcdefg.json # files created/changed within 1h of that one\n% lsx -t 5d/10d cache/*/*       # mtime in 5d-10d ago range\n% lsx -rt 2024-10-20/2024-10-25 # between those dates in the current dir\n% lsx -rt 1am/3:30 logs         # logs changed from 1am to 3:30am earlier today\n```\n\nSimple python script with no extra dependencies.\n\n\u003ca name=hdr-trunc-filenames\u003e\u003c/a\u003e\n##### [trunc-filenames](trunc-filenames)\n\nPython script to recursively shorten (truncate) file/directory names\nunder specified byte-limit, respecting typical filename format, suffixes\nand multibyte encodings.\n\nUseful for transferring files from NTFS and similar filesystems\nto POSIX/linux ones that have strict 255-byte filename-length limit,\nwhere non-english paths can get very long fast bytewise.\n\nTruncates names decoded to unicode characters to avoid splitting those,\nhas somewhat complicated rules for how to truncate filenames with dot-suffixes\nand multiple dots in them, disambiguates rename destinations on conflicts,\nalways keeps longest filename possible under `-l/--max-len` limit,\ninserts unicode-ellipsis (…) character to indicate where truncation was made.\n\nDefaults to dry-run mode for safety, only printing all renames to be made.\n\n\n\n\u003ca name=hdr-various_file-data_processing_tools\u003e\u003c/a\u003e\n#### Various file-data processing tools\n\nThings that manipulate some kind of data formats or mangle generic file/pipe contents.\n\n\u003ca name=hdr-repr\u003e\u003c/a\u003e\n##### [repr](repr)\n\nEver needed to check if file has newlines or BOM in it, yet every editor is\nuser-friendly by default and hides these from actual file contents?\n\nOne fix is hexdump or switching to binary mode, but these are usually terrible\nfor looking at text, and tend to display all non-ASCII as \".\" instead of nicer\n\\\\r \\\\t \\\\n ... escapes, not to mention unicode chars.\n\nThis trivial script prints each line in a file via python's repr(), which is\nusually very nice, has none of the above issues and doesn't dump byte codes on\nyou for anything it can interpret as char/codepoint or some neat escape code.\n\nHas opts for text/byte mode and stripping \"universal newlines\" (see newline= in\nbuilt-in open() func).\n\nCan also do encoding/newline conversion via -c option, as iconv can't do BOM or\nnewlines, and sometimes you just want \"MS utf-8 mode\" (`repr -c utf-8-sig+r`).\nUsing that with +i flag as e.g. `repr -c utf-8-sig+ri file1 file2 ...`\nconverts encoding+newlines+BOM for files in-place at no extra hassle.\n\n\u003ca name=hdr-color\u003e\u003c/a\u003e\n##### [color](color)\n\nOutputs terminal color sequences, making important output more distinctive.\n\nAlso can be used to interleave \"tail -f\" of several logfiles in the same terminal:\n\n``` console\n% t -f /var/log/app1.log | color red - \u0026\n% t -f /var/log/app2.log | color green - \u0026\n% t -f /var/log/app2.log | color blue - \u0026\n```\n\nOr to get color-escape-magic for your bash script: `color red bold p`\n\n\u003ca name=hdr-resolve-hostnames\u003e\u003c/a\u003e\n##### [resolve-hostnames](resolve-hostnames)\n\nScript (py3) to find all specified (either directly, or by regexp)\nhostnames and replace these with corresponding IP addresses, resolved\nthrough getaddrinfo(3).\n\nExamples:\n\n    % cat cjdroute.conf\n    ... \"fraggod.net:21987\": { ... },\n        \"localhost:21987\": { ... },\n        \"fraggod.net:12345\": { ... }, ...\n\n    % resolve-hostnames fraggod.net localhost \u003c cjdroute.conf\n    ... \"192.168.0.11:21987\": { ... },\n        \"127.0.0.1:21987\": { ... },\n        \"192.168.0.11:12345\": { ... }, ...\n\n    % resolve-hostnames -m '\"(?P\u003cname\u003e[\\w.]+):\\d+\"' \u003c cjdroute.conf\n    % resolve-hostnames fraggod.net:12345 \u003c cjdroute.conf\n    % resolve-hostnames -a inet6 fraggod.net localhost \u003c cjdroute.conf\n    ...\n\n    % cat nftables.conf\n    define set.gw.ipv4 = { !ipv4.name1.local, !ipv4.name2.local }\n    define set.gw.ipv6 = { !ipv6.name1.local, !ipv6.name2.local }\n    ...\n    # Will crash nft-0.6 because it treats names in anonymous sets as AF_INET (ipv4 only)\n\n    % resolve-hostnames -rum '!(\\S+\\.local)\\b' -f nftables.conf\n    define set.gw.ipv4 = { 10.12.34.1, 10.12.34.2 }\n    define set.gw.ipv6 = { fd04::1, fd04::2 }\n    ...\n\nUseful a as conf-file pre-processor for tools that cannot handle names properly\n(e.g. introduce ambiguity, can't deal with ipv4/ipv6, use weird resolvers, do it\ndynamically, etc) or should not be allowed to handle these, convert lists of\nnames (in some arbitrary format) to IP addresses, and such.\n\nHas all sorts of failure-handling and getaddrinfo-control cli options, can\nresolve port/protocol names as well.\n\n\u003ca name=hdr-resolve-conf\u003e\u003c/a\u003e\n##### [resolve-conf](resolve-conf)\n\nPython/Jinja2 script to produce a text file from a template, focused\nspecifically on templating configuration files, somewhat similar to\n\"resolve-hostnames\" above or templating provided by ansible/saltstack.\n\nJinja2 env for template has following filters and values:\n\n-   `dns(host [, af, proto, sock, default, force_unique=True])` filter/global.\n\n    getaddrinfo(3) wrapper to resolve `host` (name or address) with optional\n    parameters to a single address, raising exception if it's non-unique by default.\n\n    af/proto/sock values can be either enum value names\n    (without AF/SOL/SOCK prefix) or integers.\n\n-   `hosts` - /etc/hosts as a mapping.\n\n    For example, hosts-file line `1.2.3.4 sub.host.example.org` will produce\n    following mapping (represented as yaml):\n\n    ``` yaml\n    sub.host.example.org: 1.2.3.4\n    host.example.org:\n      sub: 1.2.3.4\n    org:\n      example:\n      host:\n        sub: 1.2.3.4\n    ```\n\n    Can be used as a reliable dns/network-independent names. `--hosts-opts`\n    cli option allows some tweaks wrt how that file is parsed.\n    See also HostsNode object for various helper methods to lookup those.\n\n-   `iface` - current network interfaces and IPv4/IPv6 addresses\n    assigned there (fetched from libc getifaddrs via ctypes).\n\n    Example value structure (as yaml):\n\n    ``` yaml\n    enp1s0:\n      - 10.0.0.134\n      - fd00::134\n      - 2001:470:1f0b:11de::134\n      - fe80::c646:19ff:fe64:632f\n    enp2s7:\n      - 10.0.1.1\n    lo:\n      - 127.0.0.1\n      - ::1\n    ip_vti0: []\n    ```\n\n    Probably a good idea to use this stuff only when IPs are static and\n    get assigned strictly before templating.\n\n-   `❴% comment_out_if value[, comment-prefix] %❵...❴% comment_out_end %❵`\n\n    (curly-braces are weird to avoid jinja2 in github-pages - replace with normal ones)\n\n    Custom template block to prefix each non-empty line within it with specified\n    string (defaults to \"#\") if value is not false-y.\n\n    Can be used when format doesn't have block comments, but it's still desirable\n    to keep disabled things in dst file (e.g. for manual tinkering) instead of\n    using if-blocks around these, or to make specific lines easier to uncomment manually.\n\n-   `it` - itertools, `zip` builtin, `szip` - `zip(a.split(), b.split())`,\n    `_v`/`v_`/`_v_` - global funcs for adding spaces before/after/around non-empty strings.\n\n-   Whatever is loaded from `--conf-file/--conf-dir` (JSON/YAML files), if specified.\n\nUse-case is a simple config file pre-processor for autonomous templating on service\nstartup with a minimal toolbox on top of jinja2, without huge dep-tree or any other\nrequirements and complexity, that is not scary to run from `ExecStartPre=` line as root.\n\n\u003ca name=hdr-temp-patch\u003e\u003c/a\u003e\n##### [temp-patch](temp-patch)\n\nTool to temporarily modify (patch) a file - until reboot or for a specified\namount of time. Uses bind-mounts from tmpfs to make sure file will be reverted\nto the original state eventually.\n\nUseful to e.g. patch `/etc/hosts` with (pre-defined) stuff from LAN on a\nlaptop (so this changes will be reverted on reboot), or a notification filter\nfile for a short \"busy!\" time period (with a time limit, so it'll auto-revert\nafter), or stuff like that.\n\nEven though dst file is mounted with \"-o ro\" by default (there's \"-w\" option to\ndisable that), linux doesn't seem to care about that option and mounts the thing\nas \"rw\" anyway, so \"chmod a-w\" gets run on temp file instead to prevent\naccidental modification (that can be lost).\n\nThere're also \"-t\" and \"-m\" flags to control timestamps during the whole\nprocess.\n\n\u003ca name=hdr-term-pipe\u003e\u003c/a\u003e\n##### [term-pipe](term-pipe)\n\nPython script with various terminal input/output piping helpers and tools.\n\nHas multiple modes for different use-cases, collected in same script mostly\nbecause they're pretty simple and not worth remembering separate ones.\n\n**out-paste**\n\nDisables terminal echo and outputs line-buffered stdin to stdout.\n\nExample use-case can be grepping through huge multiline strings\n(e.g. webpage source) pasted into terminal, i.e.:\n\n``` console\n% term-pipe | g -o '\\\u003chttps?://[^\"]\\+'\n\n[pasting page here via e.g. Shift+Insert won't cause any echo]\n\nhttps://www.w3.org/TR/html4/loose.dtd\nhttps://www.bugzilla.org/docs/3.4/en/html/bug_page.html\n...\n```\n\nThere are better tools for that particular use-case, but this solution\nis universal wrt any possible input source.\n\n**shell-notify**\n\nFilter for screen/tmux/script output to send desktop notification\n(using sd-bus lib) when shell prompt is detected on stdin, to enable when some\nlong job is running for example, so that you'd get notified immediately when it's done.\n\nShell prompt detection is done via simple regexp, highly specific to my prompt(s)\nand use-case(s), so might need tweaks in the code for different ones.\n-l/--log option can be useful when doing that - will print all input lines\n(with proper repr() wrapping), which can then be checked for desired patterns\nand tested against new detection regexps as necessary.\n\nExample use in tmux.conf:\n\n    bind-key r pipe-pane 'exec term-pipe shell-notify'\n    bind-key R pipe-pane\n\nShould make \"r\" key (after prefix key) enable notifications and \"shift+r\" disable them.\nUse \"pipe-pane -o\" to toggle this via same key instead.\n\n\"exec ...\" command there is passed to shell, so to debug errors after any\nsignificant changes, something like \"2\u003e/tmp/errors.log\" can be added at the end.\n\nCheck options of this subcommand for rate-limiting and some other tweaks.\n\n\u003ca name=hdr-yaml-to-pretty-json\u003e\u003c/a\u003e\n##### [yaml-to-pretty-json](yaml-to-pretty-json)\n\nConverts yaml files to an indented json, which is a bit more readable and\neditable by hand than the usual compact one-liner serialization.\n\nDue to yaml itself being json superset, can be used to convert json to\npretty-json as well.\n\n\u003ca name=hdr-yaml-flatten\u003e\u003c/a\u003e\n##### [yaml-flatten](yaml-flatten)\n\nConverts yaml/json files to a flat \"key: value\" lines.\n\nNested keys are flattened to a dot-separated \"level1.level2.level3\" keys,\nreplacing dots, spaces and colons there, to avoid confusing level separators\nwith the keys themselves.\n\nValues are also processed to always be one-liners, handling long values\nand empty lists/dicts and such in a readable manner too.\n\nOutput is intended for a human reader, to easily see value paths and such,\nand definitely can't be converted back to yaml or any kind of data safely.\n\n\u003ca name=hdr-yaml-diff\u003e\u003c/a\u003e\n##### [yaml-diff](yaml-diff)\n\nTool to normalize YAML files' ordering/formatting and run \"git diff | [delta]\"\non those to produce nicely-colorized and useful diffs to inspect in the terminal.\n\nLong YAMLs can be ordered and formatted in wildly different ways, and they often\nare, when produced by different tools or edited manually, hence the need for\nsomething to reformat them before running diff tools.\n\nScript can be run on two dirs to compare all yml/yaml files in those recursively\n(like \"diff -r\"), ignoring all other non-yaml files in there, as well as two\nspecific files.\n\nAlso has -f/--reformat option to pretty-print/normalize file(s) without diff,\nwhich can be used to YAML-pretty-print JSON file(s) as well (incl. recursively,\nwith --fn-re override to match them). Requires python [pygments] module to be\ninstalled for colorizing YAMLs printed to stdout with this option.\n\n\"git diff\" can be used without \"delta\" if --no-delta option is set,\nusing its own colors (as per gitconfig), but output from [delta] is usually nicer,\nhas line numbers and highlights inline diffs.\n\nBinaries and opts to both \"git diff\" and \"delta\" tools can be controlled\nvia env variables printed in -h/--help output.\n\n[delta]: https://github.com/dandavison/delta\n[pygments]: https://pygments.org/\n\n\u003ca name=hdr-hz\u003e\u003c/a\u003e\n##### [hz](hz)\n\nSame thing as the common \"head\", but works with \\\\x00\n(aka null char/byte , NUL, ␀, \\\\0, \\\\z, \\\\000, \\\\u0000, %00, ^@) delimeters.\n\nCan be done with putting \"tr\" in the pipeline before and after \"head\",\nbut this one is maybe a bit less fugly.\n\nAllows replacing input null-bytes with newlines in the output\n(--replace-with-newlines option) and vice-versa.\n\nCommon use-case is probably has something to do with filenames and xargs, e.g.:\n\n``` console\n% find -type f -print0 | shuf -z | hz -10 | xargs -0 some-cool-command\n% ls -1 | hz -z | xargs -0 some-other-command\n```\n\nI have \"h\" as an alias for \"head\" in shells, so \"head -z\" (if there were such\noption) would be aliased neatly to \"hz\", hence the script name.\n\nDefaults to reading ALL lines, not just arbitrary number (like 10, which is\ndefault for regular \"head\")!\n\n\u003ca name=hdr-liac\u003e\u003c/a\u003e\n##### [liac](liac)\n\n\"Log Interleaver And Colorizer\" python script.\n\n![interleaved colorized output][]\n\nReads lines from multiple files, ordering them by the specified field in the\noutput (default - first field, e.g. ISO8601 timestamp) and outputs each with\n(optional) unique-filename-part prefix and unique (ansi-terminal, per-file) color.\n\nMost useful for figuring out sequence of events from multiple timestamped logs.\n\nTo have safely-rotated logs with nice timestamps from any arbitrary command's\noutput, something like `stdbuf -oL \u003ccommand-and-args\u003e | svlogd -r _ -ttt \u003clog-dir\u003e`\ncan be used.\n\nNote \"stdbuf\" coreutils tool, used there to tweak output buffering, which usually\nbreaks such timestamps, and \"svlogd\" from [runit] suite (no deps, can be built separately).\n\nSee [blog post about liac tool] for more info.\n\n[interleaved colorized output]:\n  https://blog.fraggod.net/images/liac_interleaved_colorized_output.jpg\n[runit]: https://smarden.org/runit/\n[blog post about liac tool]:\n  https://blog.fraggod.net/2015/12/29/tool-to-interleave-and-colorize-lines-from-multiple-log-or-any-other-files.html\n\n\u003ca name=hdr-html-embed\u003e\u003c/a\u003e\n##### [html-embed](html-embed)\n\nScript to create \"fat\" HTML files, embedding all linked images\n(as base64-encoded data-urls), stylesheets and js into them.\n\nAll src= and href= paths must be local (e.g. \"js/script.js\" or \"/css/main.css\"),\nand will simply be treated as path components (stripping slashes on the left)\nfrom html dir, nothing external (e.g. \"//site.com/stuff.js\") will be fetched.\n\nDoesn't need anything but python, based on stdlib html.parser module.\n\nNot optimized for huge amounts of embedded data, storing all the substitutions\nin memory while it runs, and is unsafe to run on random html files, as it can\nembed something sensitive (e.g. `\u003cimg src=\"../.ssh/id_rsa\"\u003e`) - no extra checks there.\n\nUse-case is to easily produce single-file webapps or pages to pass around (or\nshare somewhere), e.g. some d3-based interactive chart page or an html report\nwith a few embedded images.\n\n\u003ca name=hdr-someml-indent\u003e\u003c/a\u003e\n##### [someml-indent](someml-indent)\n\nSimple and dirty regexp + backreferences something-ML (SGML/HTML/XML) parser to\nindent tags/values in a compact way without messing-up anything else in there.\n\nI.e. non-closed tags are FINE, something like \u003c@\u003e doesn't cause parser to\nexplode, etc.\n\nDoes not add any XML headers, does not mangle (or \"canonize\") tags/attrs/values\nin any way, except for stripping/adding those spaces.\n\nKinda like BeautifulSoup, except not limited to html and trivial enough so that\nit can be trusted not to do anything unnecessary like stuff mentioned above.\n\nFor cases when `xmllint --format` fail and/or break such kinda-ML-but-not-XML files.\n\n\u003ca name=hdr-hashname\u003e\u003c/a\u003e\n##### [hashname](hashname)\n\nScript to add simple/distinctive base32-encoded content hash to filenames.\n\nFor example:\n\n``` console\n% hashnames -p *.jpg\n\nwallpaper001.jpg -\u003e wallpaper001.kw30e7cqytmmw.jpg\nwallpaper893.jpg -\u003e wallpaper893.vbf0t0qht4dd0.jpg\nwallpaper895.jpg -\u003e wallpaper895.q5mp0j95bxbdr.jpg\nwallpaper898.jpg -\u003e wallpaper898.c9g9yeb06pdbj.jpg\n```\n\nFor collecting files with commonly-repeated names into some dir,\nlike random \"wallpaper.jpg\" or \"image.jpg\" images above from the internets.\n\nCan also be used with -t/--tag option to update names for changed files,\nwhich is handy in web-accessible dirs for changing URLs to invalidate caches.\n\nUse -h/--help for info on more useful options.\n\n\u003ca name=hdr-hhash\u003e\u003c/a\u003e\n##### [hhash](hhash.ml)\n\nProduces lower-entropy \"human hash\" phrase consisting of aspell english\ndictionary words for input arg(s) or data on stdin.\n\nIt works by first calculating BLAKE2 hash of input string/data via [libsodium],\nand then encoding it using consistent word-alphabet, exactly like something\nlike base32 or base64 does.\n\nExample:\n\n``` console\n% hhash -e AAAAC3NzaC1lZDI1NTE5AAAAIPh5/VmxDwgtJI0HiFBqZkbyV1I1YK+2DVjGjYydNp5o\nallan avenues regrade windups flours\nentropy-stats: word-count=5 dict-words=126643 word-bits=17.0 total-bits=84.8\n```\n\nHere -e is used to print entropy estimate for produced words.\n\nNote that resulting entropy values can be fractional if word-alphabet ends up\nbeing padded to map exactly to N bits (e.g. 17 bits above), so that words in it\ncan be repeated, hence not exactly 17 bits of distinct values.\n\nWritten in OCAML, linked against [libsodium] (for BLAKE2 hash function)\nvia small C glue code. Build it with:\n\n``` console\n% ocamlopt -o hhash -O2 unix.cmxa str.cmxa \\\n   -cclib -lsodium -ccopt -Wl,--no-as-needed hhash.ml hhash.ml.c\n% strip hhash\n```\n\nCaches dictionary into a ~/.cache/hhash.dict (-c option) on first run to produce\nconsistent results on this machine. Updating that dictionary will change outputs!\n\n[libsodium]: https://libsodium.org/\n\n\u003ca name=hdr-crypt\u003e\u003c/a\u003e\n##### [crypt](crypt)\n\nTrivial file/stream encryption tool using [PyNaCl's]\ncrypto_secretstream_xchacha20poly1305 authenticated encryption API.\n\nKey can be either specified on the command line for simplicity or read from a\nfile, and is always processed via scrypt, as it's likely some short string.\n\nUsage examples:\n\n``` console\n% crypt -ek my-secret-key secret.tar secret.tar.enc\n% crypt -dk my-secret-key secret.tar.enc secret.tar.test\n% crypt -ek @~/.secret.key \u003csecret.tar \u003esecret.tar.enc\n```\n\nIntended for an ad-hoc temporary encryption when transferring stuff via a usb\nstick, making a temporary backup to a random untrusted disk or whatever.\n\nDoes not support any kind of appending/resuming or partial operation, which can\nbe bad if there's a flipped bit anywhere in the encrypted data - decryption will\nstop and throw error at that point.\n\n[PyNaCl's]: https://pynacl.readthedocs.io/\n\n\n\n\u003ca name=hdr-kernel_sources_build_version_management\u003e\u003c/a\u003e\n#### Kernel sources/build/version management\n\n\u003ca name=hdr-kernel-patch\u003e\u003c/a\u003e\n##### [kernel-patch](kernel-patch)\n\nSimple stateless script to update sources in /usr/src/linux to some (specified)\nstable version.\n\nLooks for \"patch-X.Y.Z.xz\" files (as provided on kernel.org) under\n/usr/src/distfiles (configurable at the top of the script), or downloads them\nthere from kernel.org.\n\nDoes update (or rollback) by grabbing current patchset version from Makefile and\ndoing essentially `patch -R \u003c \u003cpatch-current\u003e \u0026\u0026 patch \u003c \u003cpatch-new\u003e` - i.e.\nrolling-back the current patchset, then applying new patch.\n\nAlways does `patch --dry-run` first to make sure there will be no mess left\nover by the tool and updates will be all-or-nothing.\n\nIn short, allows to run e.g. `kernel-patch 3.14.22` to get 3.14.22 in\n`/usr/src/linux` from any other clean 3.14.\\* version, or just\n`kernel-patch` to have the latest 3.14 patchset.\n\n\u003ca name=hdr-kernel-conf-check\u003e\u003c/a\u003e\n##### [kernel-conf-check](kernel-conf-check)\n\nAd-hoc python script to check any random snippet with linux kernel\n`CONFIG_...` values (e.g. \"this is stuff you want to set\" block on some wiki)\nagainst kernel config file, current config in /proc/config.gz or such.\n\nReports what matches and what doesn't to stdout, trivial regexp matching.\n\n\u003ca name=hdr-clean-boot\u003e\u003c/a\u003e\n##### [clean-boot](clean-boot)\n\nScript to remove older kernel versions (as installed by `/sbin/installkernel`)\nfrom `/boot` or similar dir.\n\nAlways keeps version linked as \"vmlinuz\", and prioritizes removal of older\npatchset versions from each major one, and only then latest per-major patchset,\nuntil free space goal (specified percentage, 20% by default) is met.\n\nAlso keeps specified number of last-to-remove versions, can prioritize cleanup\nof \".old\" verssion variants, keep `config-*` files... and other stuff (see --help).\n\nExample:\n\n    # clean-boot --debug --dry-run -f 100\n    DEBUG:root:Preserved versions (linked version, its \".old\" variant, --keep-min): 4\n    DEBUG:root: - 3.9.9.1 - System.map-3.9.9-fg.mf_master\n    DEBUG:root: - 3.9.9.1 - config-3.9.9-fg.mf_master\n    DEBUG:root: - 3.9.9.1 - vmlinuz-3.9.9-fg.mf_master\n    DEBUG:root: - 3.10.27.1 - vmlinuz-3.10.27-fg.mf_master\n    ...\n    DEBUG:root: - 3.12.19.1 - System.map-3.12.19-fg.mf_master\n    DEBUG:root: - 3.12.20.1 - config-3.12.20-fg.mf_master\n    DEBUG:root: - 3.12.20.1 - System.map-3.12.20-fg.mf_master\n    DEBUG:root: - 3.12.20.1 - vmlinuz-3.12.20-fg.mf_master\n    DEBUG:root:Removing files for version (df: 58.9%): 3.2.0.1\n    DEBUG:root: - System.map-3.2.0-fg.mf_master\n    DEBUG:root: - config-3.2.0-fg.mf_master\n    DEBUG:root: - vmlinuz-3.2.0-fg.mf_master\n    DEBUG:root:Removing files for version (df: 58.9%): 3.2.1.0\n    ... (removal of older patchsets for each major version, 3.2 - 3.12)\n    DEBUG:root:Removing files for version (df: 58.9%): 3.12.18.1\n    ... (this was the last non-latest patchset-per-major)\n    DEBUG:root:Removing files for version (df: 58.9%): 3.2.16.1\n    ... (removing latest patchset for each major version, starting from oldest - 3.2 here)\n    DEBUG:root:Removing files for version (df: 58.9%): 3.7.9.1\n    ...\n    DEBUG:root:Removing files for version (df: 58.9%): 3.8.11.1\n    ...\n    DEBUG:root:Finished (df: 58.9%, versions left: 4, versions removed: 66).\n\n(\"df\" doesn't rise here because of --dry-run, `-f 100` =\n\"remove all non-preserved\" - as df can't really get to 100%)\n\nNote how 3.2.0.1 (non-.old 3.2.0) gets removed first, then 3.2.1, 3.2.2, and so\non, but 3.2.16 (latest of 3.2.X) gets removed towards the very end, among other\n\"latest patchset for major\" versions, except those that are preserved unconditionally\n(listed at the top).\n\n\n\n\u003ca name=hdr-znc_log_helpers\u003e\u003c/a\u003e\n#### ZNC log helpers\n\nCouple scripts to manage [ZNC IRC bouncer](https://znc.in/) logs -\narchive, view, search, etc.\n\n\u003ca name=hdr-znc-log-aggregator\u003e\u003c/a\u003e\n##### [znc-log-aggregator](znc-log-aggregator)\n\nTool to process ZNC chat logs, produced by \"log\" module (one enabled globally,\nwith default wildcards) and store them using following schema under some -d/--log-dir:\n\n    \u003cnet\u003e/chat/\u003cchannel\u003e__\u003cyy\u003e-\u003cmm\u003e.log.xz\n    \u003cnet\u003e/priv/\u003cnick\u003e__\u003cyy\u003e-\u003cmm\u003e.log.xz\n\nWhere \"priv\" differs from \"chat\" in latter being prefixed by \"#\" or \"\u0026\".\n\nWith values from ZNC log paths: `moddata/log/*/\u003cnet\u003e/\u003cchan/nick\u003e/\u003cyyyy-mm-dd\u003e.log`\n\nEach ZNC-log line gets processed by regexp to add proper date, so that one\ndoesn't have to use long timestamps in ZNC itself:\n`[HH:MM:SS] \u003cnick\u003e some msg` -\u003e `[yy-mm-dd HH:MM:SS] \u003cnick\u003e some msg`.\n\nLatest (current day) logs are skipped.\nNew logs for each run are concatenated into a monthly .xz file.\n\nShould be safe to stop at any time without data loss -\nall resulting .xz's get written to temporary files and renamed at the very end,\nfollowed by unlinking of the source files, with nothing changed or updated in-place.\n\nAll temp files are produced in the destination dir, even with --dry-run,\nand should be cleaned-up on any abort/exit/finish.\n\nIdea is to have more convenient hierarchy and less files for easier shell\nnavigation/grepping (xzless/xzgrep), and without needing to worry about space\nusage of uncompressed logs in the long run.\n\nZNC changed how it stores logs a few times over the years, and this tools\nalso helped maintain consistent storage schema across these.\n\n\u003ca name=hdr-znc-log-reader\u003e\u003c/a\u003e\n##### [znc-log-reader](znc-log-reader)\n\nSame as znc-log-aggregator above, but seeks/reads specific tail (\"last n lines\")\nor time range (with additional filtering by channel/nick and network) from all\ncurrent and aggregated (via that aggregator script) ZNC logs.\n\nMostly used to query/grep recent chat logs by approximate channel name from terminal easily.\n\n\n\n\u003ca name=hdr-systemd-related\u003e\u003c/a\u003e\n#### systemd-related\n\n\u003ca name=hdr-systemd-dashboard\u003e\u003c/a\u003e\n##### [systemd-dashboard]\n\n[systemd-dashboard]: systemd-dashboard\n\nPython script to list all currently active and non-transient systemd units,\nso that these can be tracked as a \"system state\",\nand e.g. any deviations there detected/reported (simple diff can do it).\n\nGets unit info by parsing Dump() snapshot fetched via sd-bus API of libsystemd\n(using ctypes to wrap it), which is same as e.g. \"systemd-analyze dump\" gets.\n\nHas -m/--machines option to query state from all registered machines as well,\nwhich requires root (for sd_bus_open_system_machine) due to current systemd limitations.\n\nSee [Dashboard-for-... blog post] for extended rationale,\nthough it's probably obsolete otherwise since this thing was rewritten.\n\n[Dashboard-for-... blog post]:\n  https://blog.fraggod.net/2011/2/Dashboard-for-enabled-services-in-systemd\n\n\u003ca name=hdr-systemd-watchdog\u003e\u003c/a\u003e\n##### [systemd-watchdog](systemd-watchdog)\n\nTrivial script to ping systemd watchdog and do some trivial actions in-between\nto make sure os still works.\n\nWrote it after yet another silent non-crash, where linux kernel refuses to\ncreate new pids (with some backtraces) and seem to hang on some fs ops, blocking\nsyslog/journal, but leaving most simple daemons running ok-ish for a while.\n\nSo this trivial script, tied into systemd-controlled watchdog timers, tries to\ncreate pids every once in a while, with either hang or crash bubbling-up to\nsystemd (pid-1), which should reliably reboot/crash the system via hardware wdt.\n\nExample watchdog.service:\n\n``` ini\n[Service]\nType=notify\nExecStart=/usr/local/bin/systemd-watchdog -i30 -n \\\n  -f /var/log/wdt-fail.log \\\n  -x 'ip link' -x 'ip addr' -x 'ip ro' -x 'journalctl -an30'\n\nWatchdogSec=60\nTimeoutStartSec=15\nRestart=on-failure\nRestartSec=20\nStartLimitInterval=10min\nStartLimitBurst=5\nStartLimitAction=reboot-force\n\n[Install]\nWantedBy=multi-user.target\n```\n\n(be sure to tweak timeouts and test without \"reboot-force\" first though,\ne.g. pick RestartSec= for transient failures to not trigger StartLimitAction)\n\nCan optionally get IP of (non-local) gateway to 1.1.1.1 (or any specified IPv4)\nvia libmnl (also used by iproute2, so always available) and check whether it\nresponds to [fping] probes, crashing if it does not - see `-n/--check-net-gw` option.\n\nThat's mainly for remote systems which can become unreachable if kernel network\nstack, local firewall, dhcp, ethernet or whatever other link fails (usually due\nto some kind of local tinkering), ignoring more mundane internet failures.\n\nTo avoid reboot loops (in abscence of any networking), it might be a good idea\nto only start script with this option manually (e.g. right before messing up\nwith the network, or on first successful access).\n\n`-f/--fail-log` option is to log date/time of any failures for latest boot\nand run `-x/--fail-log-cmd` command(s) on any python exceptions (note: kernel\nhangs probably won't cause these), logging their stdout/stderr there -\ne.g. to dump network configuration info as in example above.\n\nUseless without systemd and requires systemd python module, plus fping tool\nif `-n/--check-net-gw` option is used.\n\n[fping]: https://fping.org/\n\n\u003ca name=hdr-cgrc\u003e\u003c/a\u003e\n##### [cgrc](cgrc)\n\nWrapper for [systemd.resource control] stuff to run commands in transient\nscopes within pre-defined slices, as well as wait for these and list pids\nwithin them easily.\n\nReplacement for things like libcgroup, cgmanager and my earlier\n[cgroup-tools project], compatible with [unified cgroup-v2 hierarchy]\nand working on top of systemd (use `systemd.unified_cgroup_hierarchy`\non cmdline, if non-default).\n\nResource limits for cgrc scopes should be defined via hierarchical slices like these:\n\n``` ini\n# apps.slice\n[Slice]\n\nCPUWeight=30\nIOWeight=30\n\nMemoryHigh=5G\nMemoryMax=8G\nMemorySwapMax=1G\n\n# apps-browser.slice\n[Slice]\nCPUWeight=30\nIOWeight=30\nMemoryHigh=3G\n```\n\nAnd then script can be used to start things there:\n\n``` console\n% cgrc apps-browser -- chromium\n% cgrc -u ff apps-browser -- firefox --profile myprofile\n```\n\nWhere e.g. last command would end up running something like this:\n\n```console\n% systemd-run -q --user --scope --unit ff \\\n  --slice apps-browser -- firefox --profile myprofile\n```\n\nNote that .scope cgroups are always transient (vanish after run), and only\n.slice ones can be pre-defined with limits.\nBoth get started/stopped by systemd on as-needed basis.\n\nTool also allows to check or list pids within scopes/slices with -c/-l options\n(to e.g. check if named scope already started or something running in a slice),\nas well as waiting on these (-q option, can be used to queue/run commands in sequence)\nand manipulating associated cgroup limits easily (-v option).\n\nRun without any args/opts or with -h/--help to get more detailed usage info.\n\n[cgroup-tools project]: https://github.com/mk-fg/cgroup-tools\n[unified cgroup-v2 hierarchy]: https://www.kernel.org/doc/Documentation/cgroup-v2.txt\n[systemd.resource control]: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html\n\n\n\n\u003ca name=hdr-ssh_and_wireguard_related\u003e\u003c/a\u003e\n#### SSH and WireGuard related\n\nSee also \"backup\" subsection.\n\n\u003ca name=hdr-ssh-fingerprint\u003e\u003c/a\u003e\n##### [ssh-fingerprint](ssh-fingerprint)\n\nssh-keyscan, but outputting each key in every possible format.\n\nImagine you have an incoming IM message \"hey, someone haxxors me, it says 'ECDSA\nkey fingerprint is f5:e5:f9:b6:a4:6b:fd:b3:07:15:f6:d9:0c:f5:47:54', what do?\",\nthis tool allows to dump any such fingerprint for a remote host, with:\n\n``` console\n% ssh-fingerprint congo.fg.nym\n...\ncongo.fg.nym ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNo...zoU04g=\n256 MD5:f5:e5:f9:b6:a4:6b:fd:b3:07:15:f6:d9:0c:f5:47:54 /tmp/.ssh_keyscan.key.kc3ur3C (ECDSA)\n256 SHA256:lFLzFQR...2ZBmIgQi/w /tmp/.ssh_keyscan.key.kc3ur3C (ECDSA)\n---- BEGIN SSH2 PUBLIC KEY ----\n...\n```\n\nOnly way I know how to get that\n\"f5:e5:f9:b6:a4:6b:fd:b3:07:15:f6:d9:0c:f5:47:54\" secret-sauce is to either do\nyour own md5 + hexdigest on ssh-keyscan output (and not mess-up due to some\nextra space or newline), or store one of the keys from there with first field\ncut off into a file and run `ssh-keygen -l -E md5 -f key.pub`.\n\nNote how \"intuitive\" it is to confirm something that ssh prints (and it prints\nonly that md5-fp thing!) for every new host you connect to with just openssh.\n\nWith this command, just running it on the remote host - presumably from diff\nlocation, or even localhost - should give (hopefully) any possible gibberish\npermutation that openssh (or something else) may decide to throw at you.\n\n\u003ca name=hdr-ssh-keyparse\u003e\u003c/a\u003e\n##### [ssh-keyparse](ssh-keyparse)\n\nPython script to extract raw private key string from ed25519 ssh keys.\n\nMain purpose is easy backup of ssh private keys and derivation of new\nsecrets from these for other purposes.\n\nFor example:\n\n``` console\n% ssh-keygen -t ed25519 -f test-key\n...\n\n% cat test-key\n-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW\nQyNTUxOQAAACDaKUyc/3dnDL+FS4/32JFsF88oQoYb2lU0QYtLgOx+yAAAAJi1Bt0atQbd\nGgAAAAtzc2gtZWQyNTUxOQAAACDaKUyc/3dnDL+FS4/32JFsF88oQoYb2lU0QYtLgOx+yA\nAAAEAc5IRaYYm2Ss4E65MYY4VewwiwyqWdBNYAZxEhZe9GpNopTJz/d2cMv4VLj/fYkWwX\nzyhChhvaVTRBi0uA7H7IAAAAE2ZyYWdnb2RAbWFsZWRpY3Rpb24BAg==\n-----END OPENSSH PRIVATE KEY-----\n\n% ssh-keyparse test-key\nHOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=\n```\n\nThat one line at the end contains 32-byte ed25519 seed (with urlsafe-base64\nencoding) - \"secret key\" - all the necessary info to restore the blob above,\nwithout extra openssh wrapping (as per PROTOCOL.key).\n\nOriginal OpenSSH format (as produced by ssh-keygen) stores \"magic string\",\nciphername (\"none\"), kdfname (\"none\"), kdfoptions (empty string), public key and\nindex for that, two \"checkint\" numbers, seed + public key string, comment and a\nbunch of extra padding at the end. All string values there are length-prefixed,\nso take extra 4 bytes, even when empty.\n\nGist is that it's a ton of stuff that's not the actual key, which ssh-keyparse\nextracts.\n\nTo restore key from seed, use `-d/--patch-key` option on any existing ed25519 key,\ne.g. `ssh-keygen -t ed25519 -N '' -f test-key \u0026\u0026 ssh-keyparse -d \u003cseed\u003e test-key`\n\nIf key is encrypted with passphrase, `ssh-keygen -p` will be run on a\ntemporary copy of it to decrypt, with a big warning in case it's not desirable.\n\nThere's also an option (`--pbkdf2`) to run the thing through PBKDF2 (tunable via\n`--pbkdf2-opts`) and various output encodings available:\n\n``` console\n% ssh-keyparse test-key  # default is urlsafe-base64 encoding\nHOSEWmGJtkrOBOuTGGOFXsMIsMqlnQTWAGcRIWXvRqQ=\n\n% ssh-keyparse test-key --hex\n1ce4845a6189b64ace04eb931863855ec308b0caa59d04d60067112165ef46a4\n\n% ssh-keyparse test-key --base32\n3KJ8-8PK1-H6V4-NKG4-XE9H-GRW5-BV1G-HC6A-MPEG-9NG0-CW8J-2SFF-8TJ0-e\n\n% ssh-keyparse test-key --base32-nodashes\n3KJ88PK1H6V4NKG4XE9HGRW5BV1GHC6AMPEG9NG0CW8J2SFF8TJ0e\n\n% ssh-keyparse test-key --raw \u003etest-key.bin\n```\n\nWith encoding like --base32 ([Douglas Crockford's human-oriented Base32],\nlast digit/lowercase-letter there is a checksum), it's easy to even read the\nthing over some voice channel, if necessary.\n\n[Douglas Crockford's human-oriented Base32]: https://www.crockford.com/wrmg/base32.html\n\n\u003ca name=hdr-ssh-key-init\u003e\u003c/a\u003e\n##### [ssh-key-init](ssh-key-init)\n\nBash script to generate (init) ssh key (via ssh-keygen) without asking about\nvarious legacy and uninteresting options and safe against replacing existing\nkeys.\n\nI.e. don't ever want RSA, ECDSA or such nonsense (Ed25519 is the norm), don't\nneed passwords for 99.999% of the keys, don't care about any of the ssh-keygen\noutput, don't need any interactivity, but do care about silently overwriting\nexisting key and want the thing to create parent dirs properly (which -f fails\nto do).\n\nHas -m option to init key for an nspawn container under `/var/lib/machines`\n(e.g. `ssh-key-init -m mymachine`) and -r option to replace any existing keys.\nSets uid/gid of the parent path for all new ones and -m700.\n\n\u003ca name=hdr-ssh-tunnel\u003e\u003c/a\u003e\n##### [ssh-tunnel](ssh-tunnel)\n\nScript to keep persistent, unique and reasonably responsive ssh tunnels.\\\nMostly just a bash wrapper with collection of options for such use-case.\n\nI.e. to run `ssh-tunnel -ti 60 2223:nexthop:22 user@host -p2222` instead of\nsome manual loop (re-)connecting every 60s in the background using something like:\n\n    ssh \\\n      -oControlPath=none -oControlMaster=no \\\n      -oConnectTimeout=5 -oServerAliveInterval=3 -oServerAliveCountMax=5 \\\n      -oPasswordAuthentication=no -oNumberOfPasswordPrompts=0 \\\n      -oBatchMode=yes -oExitOnForwardFailure=yes -TnNqy \\\n      -p2222 -L 2223:nexthop:22 user@host\n\nWhich are all pretty much required for proper background tunnel operation.\n\nHas opts for reverse-tunnels and using tping tool instead of ssh/sleep loop.\\\nKeeps pidfiles in /tmp and allows to kill running tunnel-script via same command with -k/kill appended.\n\n\u003ca name=hdr-ssh-reverse-mux-server_ssh-reverse-mux-client\u003e\u003c/a\u003e\n##### [ssh-reverse-mux-server] / [ssh-reverse-mux-client]\n[ssh-reverse-mux-server]: ssh-reverse-mux-server\n[ssh-reverse-mux-client]: ssh-reverse-mux-client\n\nPython/asyncio scripts to establish multiple ssh reverse-port-forwarding\n(\"ssh -R\") connections to the same tunnel-server from mutliple hosts using same\nexact configuration on each.\n\nNormally, first client host will bind the \"ssh -R\" listening port and all others\nwill fail, but these two scripts negotiate unique port within specified range to\neach host, so there are no clashes and all tunnels work fine.\n\nTunnel server also stores allocated ports in a db file, so that each client gets\nmore-or-less persistent listening port.\n\nEach client negotiates port before exec'ing \"ssh -R\" command, identifying itself\nvia `--ident-*` string (derived from /etc/machine-id by default), and both\nclient/server need to use same `-s/--auth-secret` to create/validate MACs in each packet.\n\nNote that all `--auth-secret` is used for is literally handing-out sequential\nnumbers, and isn't expected to be strong protection against anything, unlike ssh\nauth that should come after that.\n\n\u003ca name=hdr-wg-mux-server___wg-mux-client\u003e\u003c/a\u003e\n##### [wg-mux-server] / [wg-mux-client]\n[wg-mux-server]: wg-mux-server\n[wg-mux-client]: wg-mux-client\n\nSame thing as ssh-reverse-mux-\\* scripts above, but for negotiating WireGuard\ntunnels, with persistent host tunnel IPs tracked via `--ident-*` strings with\nsimple auth via MACs on UDP packets derived from symmetric `-s/--auth-secret`.\n\nClient identity, wg port, public key and tunnel IPs are sent in the clear with\nrelatively weak authentication (hmac of `-s/--auth-secret` string), but wg server\nis also authenticated by pre-shared public key (and `--wg-psk`, if specified).\n\nSuch setup is roughly equivalent to a password-protected (`--auth-secret`) public network.\n\nRuns \"wg set\" commands to update configuration, which need privileges,\nbut can be wrapped in sudo or suid/caps via `--wg-cmd` to avoid root in the\nrest of the script.\n\nDoes not touch or handle WireGuard private keys in any way by itself,\nand probably should not have direct access to these\n(though note that unrestricted access to \"wg\" command can reveal them anyway).\n\nExample systemd unit for server:\n\n``` console\n# wg.service + auth.secret psk.secret key.secret\n# useradd -s /usr/bin/nologin wg \u0026\u0026 mkdir -m700 ~wg \u0026\u0026 chown wg: ~wg\n# cd ~wg \u0026\u0026 cp /usr/bin/wg . \u0026\u0026 chown root:wg wg \u0026\u0026 chmod 4110 wg\n[Unit]\nWants=network.target\nAfter=network.target\n\n[Service]\nType=exec\nUser=wg\nWorkingDirectory=~\nRestart=always\nRestartSec=60\nStandardInput=file:/home/wg/auth.secret\nStandardOutput=journal\nExecStartPre=+sh -c 'ip link add wg type wireguard 2\u003e/dev/null; \\\n  ip addr add 10.123.0.1/24 dev wg 2\u003e/dev/null; ip link set wg up'\nExecStartPre=+wg set wg listen-port 1500 private-key key.secret\nExecStart=wg-mux-server --mux-port=1501 --wg-port=1500 \\\n  --wg-net=10.123.0.0/24 --wg-cmd=./wg --wg-psk=psk.secret\n\n[Install]\nWantedBy=multi-user.target\n```\n\nClient:\n\n``` console\n# wg.service + auth.secret psk.secret\n# useradd -s /usr/bin/nologin wg \u0026\u0026 mkdir -m700 ~wg \u0026\u0026 chown wg: ~wg\n# cd ~wg \u0026\u0026 cp /usr/bin/wg . \u0026\u0026 chown root:wg wg \u0026\u0026 chmod 4110 wg\n# cd ~wg \u0026\u0026 cp /usr/bin/ip . \u0026\u0026 chown root:wg ip \u0026\u0026 chmod 4110 ip\n[Unit]\nWants=network.target\nAfter=network.target\n\n[Service]\nType=exec\nUser=wg\nWorkingDirectory=~\nRestart=always\nRestartSec=10\nStandardInput=file:/home/wg/auth.secret\nStandardOutput=journal\nExecStartPre=+sh -c '[ -e key.secret ] || { umask 077; wg genkey \u003ekey.secret; }\nExecStartPre=+sh -c '[ -e key.public ] || wg pubkey \u003ckey.secret \u003ekey.public\nExecStartPre=+sh -c 'ip link add wg type wireguard 2\u003e/dev/null; ip link set wg up'\nExecStartPre=+wg set wg private-key key.secret\nExecStart=wg-mux-client \\\n  20.88.203.92:1501 BcOn/q9D5zcqK0hrWmXGQHtaEKGGf6g5nTxZUZ0P4HY= key.public \\\n  --ident-rpi --wg-net=10.123.0.0/24 --wg-cmd=./wg --ip-cmd=./ip --wg-psk=psk.secret \\\n  --ping-cmd='ping -q -w15 -c3 -i3 10.123.0.1' --ping-silent\n\n[Install]\nWantedBy=multi-user.target\n```\n\nWhen enabled, these should be enough to setup reliable tunnel up on client boot,\nand then keep it alive from there indefinitely (via `--ping-cmd` + systemd restart).\n\nExplicit iface/IP init in these units can be replaced by systemd-networkd\n.netdev + .network stuff, as it supports wireguard configuration there.\n\n\u003ca name=hdr-ssh-tunnels-cleanup\u003e\u003c/a\u003e\n##### [ssh-tunnels-cleanup](ssh-tunnels-cleanup)\n\nBash script to list or kill users' sshd pids, created for \"ssh -R\" tunnels, that\ndon't have a listening socket associated with them or don't show ssh protocol\ngreeting (e.g. \"SSH-2.0-OpenSSH_7.4\") there.\n\nThese seem to occur when ssh client suddenly dies and reconnects to create new\ntunnel - old pid can still hog listening socket (even though there's nothing on\nthe other end), but new pid won't exit and hang around uselessly.\n\nSolution is to a) check for sshd pids that don't have listenings socket, and\nb) connect to sshd pids' sockets and see if anything responds there, killing\nboth non-listening and unresponsive pids.\n\nOnly picks sshd pids for users with specific prefix, e.g. \"tun-\" by default, to\nbe sure not to kill anything useful (i.e. anything that's not for \"ssh -R\").\n\nUses ps, ss, gawk and ncat (comes with nmap), only prints pids by default\n(without `-k/--kill` option).\n\nAlso has `-s/--cleanup-sessions` option to remove all \"abandoned\" login sessions\n(think loginctl) for user with specified prefix, i.e. any leftover stuff after\nkilling those useless ssh pids.\n\nSee also: [autossh] and such.\n\n[autossh]: https://www.harding.motd.ca/autossh/\n\n\u003ca name=hdr-mosh-nat___mosh-nat-bind.c\u003e\u003c/a\u003e\n##### [mosh-nat] / [mosh-nat-bind.c]\n[mosh-nat]: mosh-nat\n[mosh-nat-bind.c]: mosh-nat-bind.c\n\nPython wrapper for mosh-server binary to do UDP hole punching through\nlocal NAT setup before starting it.\n\nComes with mosh-nat-bind.c source for LD_PRELOAD=./mnb.so lib to force\nmosh-client on the other side to use specific local port that was used in\n\"mosh-nat\".\n\nExample usage (server at 84.217.173.225, client at 74.59.38.152):\n\n``` console\nserver% ./mosh-nat 74.59.38.152\nmosh-client command:\n  MNB_PORT=34730 LD_PRELOAD=./mnb.so\n  MOSH_KEY=rYt2QFJapgKN5GUqKJH2NQ mosh-client \u003cserver-addr\u003e 34730\n\nclient% MNB_PORT=34730 LD_PRELOAD=./mnb.so \\\n  MOSH_KEY=rYt2QFJapgKN5GUqKJH2NQ mosh-client 84.217.173.225 34730\n```\n\nNotes:\n\n- mnb.so is mosh-nat-bind.c lib. Check its header for command to build it.\n- Both mnb.so and mosh-nat only work with IPv4, IPv6 shouldn't use NAT anyway.\n- Should only work like that when NAT on either side doesn't rewrite src ports.\n- 34730 is default for `-c/--client-port` and `-s/--server-port` opts.\n- Started mosh-server waits for 60s (default) for mosh-client to connect.\n- Continous operation relies on mosh keepalive packets without interruption.\n- No roaming of any kind is possible here.\n- New MOSH_KEY is generated by mosh-server on every run.\n\nUseful for direct and fast connection when there's some other means\nof access available already, e.g. ssh through some slow/indirect tunnel\nor port forwarding setup.\n\nFor more hands-off hole-punching, similar approach to what [pwnat] does can be used.\\\nSee [mobile-shell/mosh#623] for more info and links on such feature implemented in mosh directly.\\\nSource for LD_PRELOAD lib is based on \u003chttps://github.com/yongboy/bindp/\u003e\n\n[pwnat]: https://samy.pl/pwnat/\n[mobile-shell/mosh#623]: https://github.com/mobile-shell/mosh/issues/623\n\n\u003ca name=hdr-tping\u003e\u003c/a\u003e\n##### [tping](tping)\n\nPython (asyncio) tool to try connecting to specified TCP port until connection\ncan be established, then just exit, i.e. to wait until some remote port is accessible.\n\nCan be used to wait for host to reboot before trying to ssh into it, e.g.:\n\n    % tping myhost \u0026\u0026 ssh root@myhost\n\n(default `-p/--port` is 22 - ssh, see also `-s/--ssh` option)\n\nTries establishing new connection (forcing new SYN, IPv4/IPv6 should both work)\nevery `-r/--retry-delay` seconds (default: 1), only discarding (closing) \"in\nprogress\" connections after `-t/--timeout` seconds (default: 3), essentially\nkeeping rotating pool of establishing connections until one of them succeeds.\n\nThis means that with e.g. `-r1 -t5` there will be 5 establishing connections\n(to account for slow-to-respond remote hosts) rotating every second, so ratio of\nthese delays shouldn't be too high to avoid spawning too many connections.\n\nHost/port names specified on the command line are resolved synchronously on\nscript startup (same as with e.g. \"ping\" tool), so it can't be used to wait\nuntil hostname resolves, only for connection itself.\n\nAbove example can also be shortened via `-s/--ssh` option, e.g.:\n\n    % tping -s myhost 1234\n    % tping -s root@myhost:1234 # same thing as above\n    % tping -s -p1234 myhost # same thing as above\n\nWill exec `ssh -p1234 root@myhost` immediately after successful tcp connection.\n\nUses python stdlib stuff, namely asyncio, to juggle multiple connections\nin an efficient manner.\n\n\n\n\u003ca name=hdr-wifi___bluetooth_helpers\u003e\u003c/a\u003e\n#### WiFi / Bluetooth helpers\n\n\u003ca name=hdr-adhocapd\u003e\u003c/a\u003e\n##### [adhocapd](adhocapd)\n\nPicks first wireless dev from `iw dev` and runs [hostapd] + udhcpd\n(from busybox) on it.\n\nConfiguration for both is generated using reasonable defaults - distinctive\n(picked from `ssid_list` at the top of the script) AP name and\nrandom password (using `passgen` from this repo or falling back to\n`tr -cd '[:alnum:]' \u003c/dev/urandom | head -c10`).\n\nDev, ssid, password, ip range and such can also be specified on the\ncommand line (see `--help`).\n\nIf inet access thru local machine is needed, don't forget to also do something\nlike this (with default ip range of 10.67.35.0/24 and \"wlp0s18f2u2\" interface name):\n\n    # sysctl -w net.ipv4.conf.all.forwarding=1\n    # iptables -t nat -A POSTROUTING -s 10.67.35.0/24 -j MASQUERADE\n    # iptables -A FORWARD -s 10.67.35.0/24 -i wlp0s18f2u2 -j ACCEPT\n    # iptables -A FORWARD -d 10.67.35.0/24 -o wlp0s18f2u2 -j ACCEPT\n\nThese rules are also echoed in the script, with IP and interface name that was\nused.\n\nFor consistent naming of network interfaces from usb devices (to e.g.  have\nconstant set of firewall rules for these), following udev rule can be used (all\nusb-wlan interfaces will be named according to NAME there):\n\n    SUBSYSTEM==\"net\", ACTION==\"add\", ENV{DEVTYPE}==\"wlan\",\\\n      DEVPATH==\"*/usb[0-9]/*\", NAME=\"wlan_usb\"\n\n[hostapd]: https://w1.fi/hostapd/\n\n\u003ca name=hdr-wpa-systemd-wrapper\u003e\u003c/a\u003e\n##### [wpa-systemd-wrapper](wpa-systemd-wrapper)\n\nSystemd wrapper for [wpa_supplicant] or [hostapd], enabling either to\nwork with Type=notify, support WatchdogSec=, different exit codes and\nall that goodness.\n\nStarts the daemon as a subprocess, connecting to its management interface and\nwatching state/wpa_state changes, only indicating \"started\" state for systemd\nwhen daemon actually starts scanning/connecting (for wpa_supplicant) or sets\nstate=enabled for hostapd.\n\nWatchdogSec= issues PING commands to underlying daemon, proxying responses back,\nas long as daemon state is somehting valid, and not INTERFACE-DISABLED,\nlocally-generated disconnect or such, usually indicating hw failure, kernel\nmodule issue or whatever else.\n\nSuch thing is needed to have systemd unit state follow AP/STA state, failing\nwhen e.g. wifi dongle gets pulled out from USB port, as that doesn't actually\ncause these things to fail/exit otherwise, which might be desirable if that wifi\nlink is critical to other services or as a reboot-workaround for driver bugs.\n\nExample systemd unit (AP mode):\n\n``` ini\n[Service]\nExecStart=/usr/local/bin/wpa-systemd-wrapper \\\n  --exit-check '/run/wpa.wlan0.first-run:config' \\\n  --ap-mode wlan0 /etc/hostapd.wlan0.conf\n\nType=notify\nWatchdogSec=90\nRestart=on-failure\nRestartPreventExitStatus=78\nRestartSec=3\n# StartLimitInterval=8min\n# StartLimitBurst=10\n# StartLimitAction=reboot\n```\n\nThis will run hostapd (due to `-a/--ap-mode`), and exit with special 78/CONFIG\ncode if \"first-run\" file exists and hostapd never gets into ENABLED state on the\nfirst attempt - i.e. something likely wrong with the config and there's no point\nrestarting it ad nauseum.\n\nPython/asyncio, requires python-systemd installed, use `-h/--help`\nand `-d/--debug` opts for more info.\n\n[wpa_supplicant]: https://w1.fi/wpa_supplicant/\n\n\u003ca name=hdr-timed-ble-beacon\u003e\u003c/a\u003e\n##### [timed-ble-beacon](timed-ble-beacon)\n\nPython script to broadcast [Bluetooth Low Energy (BLE)] beacons\nfor specified amount of time, with a time countdown in them,\nusing standard linux [BlueZ bluetooth stack].\n\nBroadcasts are done using Primary Advertising mechanism (ADV\\_SCAN\\_IND PDUs),\nnot marked as \"discoverable\", intended be picked-up by passive scans on recipient.\nAll data is embedded in \"Manufacturer Specific Data\" bytes, where in addition to\ncountdown, there's also replay counter and keyed HMAC, to prevent replays or a\nsimple forgery.\n\nBoth sender (broadcaster) and recipient (observer) should share configured keys\nfor communication to work.\n\nIntended to be used to temporarily enable/disable something while BLE beacons\nare being broadcast, receiving/checking those on cheap [micropython] controllers -\nkinda like a smart-home remotely-controlled switch, but automatically reverting\nto default state on its own, standalone, and much simpler.\n\n[timed-ble-beacon-mpy-led micropython script] is the receiver side,\nintended to run on a cheap [RPi Pico W] board with rp2040 microcontroller.\nThere can be any number of senders/receivers at the same time - just use\ndifferent `--mid` and `-s/--secret` values for different control-domains.\n\nCan be debug-run like this: `./timed-ble-beacon -dt 5m`\n\nUses [python-dbus] and [python-gobject] modules to work\nwith bluez over dbus within glib eventloop.\n\n[BlueZ bluetooth stack]: https://www.bluez.org/\n[Bluetooth Low Energy (BLE)]: https://en.wikipedia.org/wiki/Bluetooth_Low_Energy\n[micropython]: https://docs.micropython.org/en/latest/\n[RPi Pico W]: https://www.raspberrypi.com/documentation/microcontrollers/pico-series.html#picow-technical-specification\n[timed-ble-beacon-mpy-led micropython script]: #hdr-timed-ble-beacon-mpy-led\n[python-dbus]: https://dbus.freedesktop.org/doc/dbus-python/\n[python-gobject]: https://pygobject.gnome.org/\n\n\u003ca name=hdr-timed-ble-beacon-mpy-led\u003e\u003c/a\u003e\n##### [timed-ble-beacon-mpy-led](timed-ble-beacon-mpy-led)\n\nMicopython script to passively scan for [Bluetooth Low Energy (BLE)] beacons\nwith specific HMAC-authenticated payload, and trigger some action while those\nare active, reverting back to default state otherwise.\n\nIntended to run on [RPi Pico W] or [ESP32] microcontrollers, or anything else\nsupported by [micropython], and default code just blinks connected LEDs in a\nconfigured pattern, as an indicator/notification task.\n\n[timed-ble-beacon script] above can be used to broadcast BLE beacons in question.\nMust be configured with at least mid/key parameters when calling it via import +\nrun_with_conf() or at the top of the script, unless just testing with defaults\nin both of these.\n\nTo setup/run this on a usb-tty-connected microcontroller board:\n\n``` console\n## Upload micropython firmware to the device, install \"mpremote\" tool\n\n% nano timed-ble-beacon-mpy-led\n## Set parameters like ble_mid and ble_secret at the top of the script\n\n% mpremote run timed-ble-beacon-mpy-led\n## Should either work or print some errors to console\n\n## To setup this script to run on board boot with all-hardcoded parameters\n% mpremote cp timed-ble-beacon-mpy-led :main.py\n% mpremote reset\n\n## Alternatively, configuration can be provided via loader-script\n% mpremote cp timed-ble-beacon-mpy-led :tbbml.py\n% echo \u003emain.py 'import tbbml'\n% echo \u003e\u003emain.py 'tbbml.run_with_conf(verbose=1, led_pin=2, ble_mid=123, ble_secret=b\"test\")'\n% mpremote cp main.py :\n\n## Can also compile \"tbbml\" for loader-script via mpy-cross for size/mem/load-times\n% mpy-cross -march=armv6m -O2 timed-ble-beacon-mpy-led -o tbbml.mpy\n% mpremote cp tbbml.mpy :\n```\n\nAction-task in this script simply blinks LED indicator (e.g. built-in\n`machine.Pin('LED')` by default, can use multiple LEDs) with randomized\nintervals when no beacons are detected.\n\nSee Conf class in this, as well as timed-ble-beacon script above\nand its `-h/--help` output for more details.\n\n[ESP32]: https://en.wikipedia.org/wiki/ESP32\n[timed-ble-beacon script]: #hdr-timed-ble-beacon\n\n\n\n\u003ca name=hdr-misc\u003e\u003c/a\u003e\n#### Misc\n\nMisc one-off scripts that don't group well with anythin else.\n\n\u003ca name=hdr-at\u003e\u003c/a\u003e\n##### [at](at)\n\nReplacement for standard unix'ish \"atd\" daemon in the form of a bash script.\n\nIt just forks out and waits for however long it needs before executing the given command.\\\nUnlike atd proper, such tasks won't survive reboot, obviously.\n\n    Usage: ./at [ -h | -v ] when \u003c sh_script\n    With -v flag ./at mails script output if it's not empty even if exit code is zero.\n\n\u003ca name=hdr-sleepc\u003e\u003c/a\u003e\n##### [sleepc](sleepc)\n\nPython script that works like a verbose \"sleep\" tool - prints countdown until\nspecified time to terminal, and also parses more wide variety of relative/absolute\ntimestamp formats:\n\n``` console\n% ./sleepc 3h2m\nParsed time-spec '3h2m' as 2023-06-11 23:10:12.459720 [in 3h 2m]\nCountdown: 3:01:59 [in 3h 2m]\n```\n\nUseful for waiting with a known time or delay in interactive consoles, to avoid\nneeding to calculate offset for \"sleep\", and be able to check back on it later.\n\n\u003ca name=hdr-wgets\u003e\u003c/a\u003e\n##### [wgets](wgets)\n\nSimple script to grab a file using wget and then validate checksum of\nthe result, e.g.:\n\n``` console\n$ wgets -c https://os.archlinuxarm.org/os/ArchLinuxARM-sun4i-latest.tar.gz cea5d785df19151806aa5ac3a917e41c\n\nUsing hash: md5\nUsing output filename: ArchLinuxARM-sun4i-latest.tar.gz\n--2014-09-27 00:04:45--  https://os.archlinuxarm.org/os/ArchLinuxARM-sun4i-latest.tar.gz\nResolving os.archlinuxarm.org (os.archlinuxarm.org)... 142.4.223.96, 67.23.118.182, 54.203.244.41, ...\nConnecting to os.archlinuxarm.org (os.archlinuxarm.org)|142.4.223.96|:80... connected.\nHTTP request sent, awaiting response... 416 Requested Range Not Satisfiable\n\n    The file is already fully retrieved; nothing to do.\n\nChecksum matched\n```\n\nBasic invocation syntax is `wgets [ wget_opts ] url checksum`,\nchecksum is hex-decoded and hash func is auto-detected from its length\n(md5, sha-1, all sha-2's are supported).\n\nIdea is that - upon encountering an http link with either checksum on the page\nor in the file nearby - you can easily run the thing providing both link and\nchecksum to fetch the file.\n\nIf checksum is available in e.g. \\*.sha1 file alongside the original one,\nit might be a good idea to fetch that checksum on a different host or a proxy,\nmaking spoofing of both checksum and the original file on the same connection\na bit harder.\n\n\u003ca name=hdr-mail\u003e\u003c/a\u003e\n##### [mail](mail)\n\nSimple bash wrapper for sendmail command, generating From/Date headers and\nstuff, just like mailx would do, but also allowing to pass custom headers\n(useful for filtering error reports by-source), which some implementations\nof \"mail\" fail to do.\n\n\u003ca name=hdr-passgen\u003e\u003c/a\u003e\n##### [passgen](passgen)\n\nUses aspell english dictionaly to generate easy-to-remember passphrase -\na [Diceware-like] method.\n\n  [Diceware-like]: https://en.wikipedia.org/wiki/Diceware\n\nUse -e option to get a rough entropy estimate for the resulting passphrase,\nbased on number of words in aspell dictionary dump that is being used.\n\nOther options allow for picking number of words and sanity-checks like min/max\nlength (to avoid making it too unwieldy or easy to bruteforce via other methods).\n\n\u003ca name=hdr-urlparse\u003e\u003c/a\u003e\n##### [urlparse](urlparse)\n\nSimple script to parse long URL with lots of parameters, decode and print it out\nin an easily readable ordered YAML format or diff (that is, just using \"diff\"\ncommand on two outputs) with another URL.\n\nNo more squinting at some huge incomprehensible ecommerce URLs before scraping\nthe hell out of them!\n\n\u003ca name=hdr-ip-ext\u003e\u003c/a\u003e\n##### [ip-ext](ip-ext)\n\nSome minor tools for network configuration from console/scripts, which iproute2\nseem to be lacking, in a py3 script.\n\nFor instance, if network interface on a remote machine was (mis-)configured in\ninitramfs or wherever to not have link-local IPv6 address, there seem to be no\ntool to restore it without whole \"ip link down \u0026\u0026 ip link up\" dance, which can\nbe a bad idea.\n\n`ipv6-lladdr` subcommand handles that particular case, generating ipv6-lladdr\nfrom mac, as per RFC 4291 (as implemented in \"netaddr\" module) and can assign\nresulting address to the interface, if missing:\n\n``` console\n# ip-ext --debug ipv6-lladdr -i enp0s9 -x\nDEBUG:root:Got lladdr from interface (enp0s9): 00:e0:4c:c2:78:86\nDEBUG:root:Assigned ipv6_lladdr (fe80::2e0:4cff:fec2:7886) to interface: enp0s9\n```\n\n`ipv6-dns` tool generates \\*.ip.arpa and djbdns records for specified IPv6.\n\n`ipv6-name` encodes or hashes name into IPv6 address suffix to produce\nan easy-to-remember static ones.\n\n`iptables-flush` removes all iptables/ip6tables rules from all tables,\nincluding any custom chains, using iptables-save/restore command-line tools,\nand sets policy for default chains to ACCEPT.\n\n\u003ca name=hdr-blinky\u003e\u003c/a\u003e\n##### [blinky](blinky)\n\nScript to blink gpio-connected leds via `/sys/class/gpio` interface.\n\nIncludes oneshot mode, countdown mode (with some interval scaling option),\ndirect on-off phase delay control (see --pre, --post and --interval\\* options),\ncooperation between several instances using same gpio pin, \"until\" timestamp\nspec, and generally everything I can think of being useful (mostly for use from\nother scripts though).\n\n\u003ca name=hdr-openssl-fingerprint\u003e\u003c/a\u003e\n##### [openssl-fingerprint](openssl-fingerprint)\n\nDo `openssl s_client -connect somesite \u003c/dev/null | openssl x509 -fingerprint -noout -sha1`\nin a nicer way - openssl cli tool doesn't seem to have that.\n\nAlso can be passed socks proxy IP:PORT to use socat and pipe openssl\nconnection through it - for example, to get fingerprint over Tor\n(with `SocksAddress localhost:1080`) link:\n\n``` console\n% openssl-fingerprint google.com localhost:1080\nSHA1 Fingerprint=A8:7A:93:13:23:2E:97:4A:08:83:DD:09:C4:5F:37:D5:B7:4E:E2:D4\n```\n\n\u003ca name=hdr-nsh\u003e\u003c/a\u003e\n##### [nsh](nsh)\n\nBash script to \"nsenter\" into specified machine's (as can be seen in\n`ps -eo machine` or `nsh` when run without args) container namespaces\nand run login shell there.\n\nMachine in question must run systemd as pid-1 (e.g. systemd-nspawn container),\nas it gets picked as `--target` pid for nsenter.\n\nVery similar to `machinectl login \u003cmachine\u003e`, but does not asks for\nuser/password and does not start new `systemd --user` session,\njust runs `su -` to get root login shell.\n\nEssentially same as `machinectl shell \u003cmachine\u003e`, but doesn't require\nsystemd-225+ and machine being registered with systemd at all.\n\nIf running `tty` there says `not a tty` and e.g. `screen` bails out with\n`Must be connected to a terminal.`, just run extra `getty tty` there - will\nask to login (be mindful of /etc/securetty if login fails), and everything\ntty-related should work fine afterwards.\n\nIf run without argument or with `-l/--list` option, will list running machines.\n\nSee also: lsns(1), nsenter(1), unshare(1)\n\n\u003ca name=hdr-pam-run\u003e\u003c/a\u003e\n##### [pam-run](pam-run)\n\nWrapper that opens specified PAM session (as per one of the configs in\n`/etc/pam.d`, e.g. \"system-login\"), switches to specified uid/gid and runs\nsome command there.\n\nMy use-case is to emulate proper \"login\" session for systemd-logind, which\nneither \"su\" nor \"sudo\" can do (nor should do!) in default pam configurations\nfor them, as they don't load pam_systemd.so (as opposed to something like\n`machinectl shell myuser@ -- ...`).\n\nThis script can load any pam stack however, so e.g. running it as:\n\n    # pam-run -s system-login -u myuser -t :1 \\\n      -- bash -c 'systemctl --user import-environment \\\n        \u0026\u0026 systemctl --user start xorg.target \u0026\u0026 sleep infinity'\n\nShould initiate proper systemd-logind session (and close it afterwards)\nand start \"xorg.target\" in \"myuser\"-specific \"systemd --user\" instance\n(started by logind with the session).\n\nCan be used as a GDM-less way to start/keep such sessions (with proper\ndisplay/tty and class/type from env) without much hassle or other weirdness\nlike \"agetty --autologin\" or \"login\" in some pty (see also [mk-fg/de-setup] repo),\nor for whatever other pam wrapping or testing (e.g. try logins with passwords\nfrom file), as it has nothing specific (or even related) to desktops.\n\nSelf-contained python script, using libpam via ctypes.\n\nWarning: this script is no replacement for su/sudo wrt uid/gid-switching, and\ndoesn't implement all the checks and sanitization these tools do, so only\nintended to be run from static, clean or trusted environment (e.g. started by\nsystemd or manually).\n\n[mk-fg/de-setup]: https://github.com/mk-fg/de-setup\n\n\u003ca name=hdr-primes\u003e\u003c/a\u003e\n##### [primes](primes)\n\nPython script to print prime numbers in specified range.\n\nFor small ranges only, as it does dumbest brute-force \\[2, sqrt(n)\\] division checks,\nand intended to generate primes for non-overlapping \"tick % n\" workload spacing,\nnot any kind of crypto operations.\n\n\u003ca name=hdr-boot-patcher\u003e\u003c/a\u003e\n##### [boot-patcher](boot-patcher)\n\nPy script to run on early boot, checking specific directory for update-files\nand unpack/run these, recording names to skip applied ones on subsequent boots.\n\nIdea for it is to be very simple, straightforward, single-file drop-in script to\nput on distributed .img files to avoid re-making these on every one-liner change,\nsending tiny .update files instead.\n\nUpdate-file format:\n\n- Either zip or bash script with .update suffix.\n- Script/zip detected by python's zipfile.is_zipfile() (zip file magic).\n- If zip, should contain \"_install\" (update-install) script inside.\n- Update-install script shebang is optional, defaults to \"#!/bin/bash\".\n\nUpdate-install script env:\n\n- BP_UPDATE_ID: name of the update (without .update suffix, e.g. \"001.test\").\n\n- BP_UPDATE_DIR: unpacked update zip dir in tmpfs.\n\n    Will only have \"_install\" file in it for standalone scripts (non-zip).\n\n- BP_UPDATE_STATE: /var/lib/boot-patcher/\u003cupdate-id\u003e\n\n    Persistent dir created for this update, can be used to backup various\n    updated/removed files, just in case.\n    If left empty, removed after update-install script is done.\n\n- BP_UPDATE_STATE_ROOT: /var/lib/boot-patcher\n\n- BP_UPDATE_REBOOT: reboot-after flag-file (on tmpfs) to touch.\n\n    If reboot is required after this update, create (touch) file at that path.\\\n    Reboot will be done immediately after this particular update, not after all of them.\n\n- BP_UPDATE_REAPPLY: flag-file (on tmpfs) to re-run this update on next boot.\n\n    Can be used to retry failed updates by e.g. creating it at the start of the\n    script and removing on success.\n\nExample update-file contents:\n\n-   2017-10-27.001.install-stuff.zip.update\n\n    `_install`:\n\n        cd \"$BP_UPDATE_DIR\"\n        exec pacman --noconfirm -U *.pkg.tar.xz\n\n    `*.pkg.tar.xz` - any packages to install, zipped alongside that ^^^\n\n-   2017-10-28.001.disable-console-logging.update (single update-install file):\n\n        patch -l /boot/boot.ini \u003c\u003c'EOF'\n        --- /boot/boot.ini.old  2017-10-28 04:11:15.836588509 +0000\n        +++ /boot/boot.ini      2017-10-28 04:11:38.000000000 +0000\n        @@ -6,7 +6,7 @@\n         hdmitx edid\n\n         setenv condev \"console=ttyAML0,115200n8 console=tty0\"\n        -setenv bootargs \"root=/dev/mmcblk1p2 ... video=HDMI-A-1:1920x1080@60e\"\n        +setenv bootargs \"root=/dev/mmcblk1p2 ... video=HDMI-A-1:1920x1080@60e loglevel=1\"\n\n         setenv loadaddr \"0x1080000\"\n         setenv dtb_loadaddr \"0x1000000\"\n        EOF\n        touch \"$BP_UPDATE_REBOOT\"\n\n-   2017-10-28.002.apply-patches-from-git.zip.update\n\n    `_install`:\n\n        set -e -o pipefail\n        cd /srv/app\n        for p in \"$BP_UPDATE_DIR\"/*.patch ; do patch -p1 -i \"$p\"; done\n\n    `*.patch` - patches for \"app\" from the repo, made by e.g. `git format-patch -3`.\n\nMisc notes:\n\n- Update-install exit code is not checked.\n\n- After update-install is finished, and if BP_UPDATE_REAPPLY was not created,\n  \"\u003cupdate-id\u003e.done\" file is created in BP_UPDATE_STATE_ROOT and update is\n  skipped on all subsequent runs.\n\n- Update ordering is simple alphasort, dependenciess can be checked by update\n  scripts via .done files (also mentioned in prev item).\n\n- No auth (e.g. signature checks) for update-files, so be sure to send these\n  over secure channels.\n\n- Run as `boot-patcher --print-systemd-unit` for the only bit of setup it needs.\n\n\u003ca name=hdr-audit-follow\u003e\u003c/a\u003e\n##### [audit-follow](audit-follow)\n\nSimple py3 script to decode audit messages from \"journalctl -af -o json\" output,\ni.e. stuff like this:\n\n    Jul 24 17:14:01 malediction audit: PROCTITLE\n      proctitle=7368002D630067726570202D652044... (loooong hex-encoded string)\n    Jul 24 17:14:01 malediction audit: SOCKADDR saddr=020000517F0000010000000000000000\n\nInto this:\n\n    PROCTITLE proctitle='sh -c grep -e Dirty: -e Writeback: /proc/meminfo'\n    SOCKADDR saddr=127.0.0.1:81\n\nFilters for audit messages only, strips long audit-id/time prefixes,\nunless -a/--all specified, puts separators between multi-line audit reports,\nrelative and/or differential timestamps (-r/--reltime and -d/--difftime opts).\n\nAudit subsystem can be very useful to understand which process modifies some\npath, what's the command-line of some /bin/bash being run from somewhere\noccasionally, or what process/command-line connects to some specific IP and what\nscripts it opens beforehand - all without need for gdb/strace, or where they're\ninapplicable.\n\nSome useful incantations (cheatsheet):\n\n    # auditctl -e 1\n    # auditctl -a exit,always -S execve -F path=/bin/bash\n    # auditctl -a exit,always -F auid=1001 -S open -S openat\n    # auditctl -w /some/important/path/ -p rwxa\n    # auditctl -a exit,always -F arch=b64 -S connect\n\n    # audit-follow -ro='--since=-30min SYSLOG_IDENTIFIER=audit' |\n      grep --line-buffered -B1000 -F some-interesting-stuff | tee -a audit.log\n\n    # auditctl -e 0\n    # auditctl -D\n\nauditd + ausearch can be used as an offline/advanced alternative to such script.\\\nMore powerful options for such task on linux can be sysdig and various BPF tools.\n\n\u003ca name=hdr-tui-binary-conv\u003e\u003c/a\u003e\n##### [tui-binary-conv](tui-binary-conv)\n\nSimple ncurses-based interactive (TUI) decimal/hex/binary\npy3 converter script for the terminal.\n\nMain purpose it to easily experiment with flipping bits and digits in values,\nseeing nicely aligned/formatted/highlighted immediate changes in other outputs\nand an easy converter tool as well.\n\nControls are: cursor keys, home/end, backspace, insert (insert/replace mode),\n0/1 + digits + a-f, q to quit.\n\nThere's a picture of it [on the blog page here].\n\n[on the blog page here]:\n  https://blog.fraggod.net/2019/01/10/tui-console-dechexbinary-converter-tool.html\n\n\u003ca name=hdr-maildir-cat\u003e\u003c/a\u003e\n##### [maildir-cat](maildir-cat)\n\nPython script to iterate over all messages in all folders of a maildir and\nprint (decoded) headers and plain + html body of each (decoded) message, with\nevery line prefixed by its filename.\n\nIntended use is to produce a text dump of a maildir for searching or processing\nit via any simple tools like grep or awk.\n\nSo using e.g. `maildir-cat | grep 'important-word'` will produce same output\nas `grep -r 'important-word' email-texts/` would if emails+headers were dumped\nas simple text files there.\n\nCan also be pointed to maildir subdirs (same thing) or individual files.\\\nUses python stdlib email.* modules for all processing.\n\n\u003ca name=hdr-dns-update-proxy\u003e\u003c/a\u003e\n##### [dns-update-proxy](dns-update-proxy)\n\nSmall py3/asyncio UDP listener that receives ~100B `pk || box(name:addr)`\nlibnacl-encrypted packets, decrypts (name, addr) tuples from there,\nchecking that:\n\n- Public key of the sender is in `-a/--auth-key` list.\n- Name doesn't resolve to same IP already, among any others (`-c/--check` option).\n- Name has one of the allowed domain suffixes (`-d/--update` option).\n\nIf all these pass, specified BIND-format zone-file (for e.g. [nsd]) is updated,\nor DNS service API used to same effect, with several retries on any fails\n(`-r/--retry` option) and rate-limiting, as well as `--debug` logging.\n\nUseful wrapper for auto-updating names in delegated nsd-managed zone,\nor doing same via DNS APIs that only provide all-or-nothing access,\nwhile you want to setup convenience names from some shared-access VM,\nwithout giving away creds for the whole account on these services,\nwith all other names and subdomains there.\n\nExample snippet for sending update packets:\n\n``` python\nimport socket, time, libnacl.public, base64, pathlib as pl\n\nb64_decode = lambda s: ( base64.urlsafe_b64decode\n  if '-' in s or '_' in s else base64.standard_b64decode )(s)\n\nclass Conf:\n  proxy_addr = 'dns-proxy.host.net'\n  proxy_pk = 'wnQvfuzUNyjDgFhPa23y0z5iXJl8TuZ+rdL0G3vefxQ='\n  sk_file = 'local_key.secret' # use e.g. \"wg genkey\" or libnacl\n  key = libnacl.public.SecretKey(b64_decode(pl.Path(sk_file).read_text()))\n  box = libnacl.public.Box(key, b64_decode(proxy_pk))\n  encrypt = lambda s, msg: s.key.pk + s.box.encrypt(msg)\nproxy_conf = Conf()\n\ndef update_dns(conf, name, addr):\n  msg = conf.encrypt(f'{name}:{addr}'.encode())\n  with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:\n  for delay in [0.1, 0.5, 1, 3, 0]:\n    try: s.sendto(msg, conf.proxy_addr)\n    except (socket.gaierror, socket.error): pass\n    if delay: time.sleep(delay)\n\nupdate_dns(proxy_conf, 'my.ddns.host.net', '1.2.3.4')\n```\n\n[nsd]: https://wiki.alpinelinux.org/wiki/Setting_up_nsd_DNS_server\n\n\u003ca name=hdr-dns-test-daemon\u003e\u003c/a\u003e\n##### [dns-test-daemon](dns-test-daemon)\n\nPython + [async_dns] authoritative DNS resolver daemon to\nreturn hashed-name results for testing DNS resolver operation.\n\nFor example:\n\n``` console\n% ./dns-test-daemon -k hash-key -b 127.0.0.1:5533 \u0026\n% dig -p5533 @127.0.0.1 aaaa test.com\n...\ntest.com. 300 IN AAAA eb5:7823:f2d2:2ed2:ba27:dd79:a33e:f762\n...\n```\n\nHere, for AAAA \"test.com\" query, script returned first 16 bytes of\n\"blake2s(test.com, key=hash-key, person=dnstd.1)\" hash digest\nas a reponse (converted to address via inet_ntop).\n\nIts purpose is to be run as an authoritative resolver for some stub zone\nforwarded to it, e.g. \"\\*.test.mydomain.com\", and then be able to make sure that\nany local DNS resolver works by querying e.g. \"12345.test.mydomain.com\" and\nchecking that resulting address hash matches expected value (dependent only on\nqueried name, hash key and that hardcoded person= string).\n\nTo run script in tester-client mode, simply pass it a name to test, along with\nsame `-k/--hash-key` parameter as for daemon on the other end, e.g.:\n\n    % ./dns-test-daemon -k hash-key random-stuff.test.mydomain.com\n    % ./dns-test-daemon -k hash-key --debug @.test.mydomain.com\n\nIt will exit with non-zero code if result is missing or doesn't match expected\nvalue in any way.\n\nDoes not import/use or require asyncio and async_dns modules in client mode.\n\nIts `-c/--continuous` mode can be used together with systemd to kick/restart\nunreliable resolver daemon (e.g. unbound) when it hangs or fails in other ways:\n\n``` ini\n[Service]\nType=exec\nUser=dnstd\nExecStart=dns-test-daemon -c 150:6:100 -p 1.1.1.1 @.test.mydomain.com\nExecStopPost=+bash -c '[[ \"$$SERVICE_RESULT\" = success ]] || systemctl try-restart unbound'\n\n# Using RestartForceExitStatus=53 should prevent unbound restarts on script bugs\nRestartForceExitStatus=53\nRestartSec=5min\n\n[Install]\nWantedBy=multi-user.service\n```\n\nNote `-p 1.1.1.1` ping-option there to avoid restarting the daemon if whole\nnetwork is down, which runs \"fping\" to check that on detected DNS failures.\n\n[async_dns]: https://github.com/gera2ld/async_dns\n\n\u003ca name=hdr-nginx-access-log-stat-block\u003e\u003c/a\u003e\n##### [nginx-access-log-stat-block](nginx-access-log-stat-block)\n\nPython/ctypes script to be used alongside [nginx-stat-check] module, reliably\ntailing any kind of access.log-like file(s) where first (space-separated) field\nis IP address and creating files with name corresponding to these in specified db_dir.\n\nnginx-stat-check module then allows to use `stat_check /some/db_dir/$remote_addr;`\nin nginx.conf to return 403 for all addresses processed in this way.\n\nCreated files are automatically renamed and cleaned-up after specified\nunblock/forget-timeouts and block-timeout either get extended or multiplied by\nspecified k value (2x default) on repeated blocks after expiry.\n\nIntended use it to block stupid bots and whatever spammers that don't care about\nrobots.txt when these access some honeypot-file on nginx level (with proper 403\non specific URL paths), which normally should never be requested.\n\nI.e. bots that are stupidly re-indexing giant file dumps or whatever dynamic\ncontent every N minutes.\n\nExample nginx.conf snippet:\n\n    load_module /usr/lib/nginx/modules/ngx_http_stat_check.so;\n    log_format stat-block '$remote_addr :: $time_iso8601 \"$http_referer\" \"$http_user_agent\"';\n    ...\n\n    location = /distro/package/mirror/open-and-get-banned.txt {\n      alias /srv/pkg-mirror/open-and-get-banned.txt;\n      access_log /var/log/nginx/bots.log stat-block;\n    }\n\n    location /distro/package/mirror {\n      alias /srv/pkg-mirror;\n      autoindex on;\n      stat_check /tmp/stat-block/$remote_addr;\n    }\n\nAnd run script to populate `/tmp/stat-block/` path from bots.log:\n\n    % ./nginx-access-log-stat-block --debug /tmp/stat-block/ /var/log/nginx/bots.log\n\nCheck `-h/--help` output for default block-timeout and such values.\n\nUses inotify to tail files via ctypes, detects log rotation but NOT truncation\n(use with append/remove-only logs), can tail multiple wildcard-matching files\nin a directory, closes opened/tailed logs after timeout.\n\nAlways opens files at the end, so can loose a line or two due to that,\nwhich is fine for intended purpose (bots spam requests anyway).\n\n[nginx-stat-check]: https://github.com/mk-fg/nginx-stat-check\n\n\u003ca name=hdr-sys-wait\u003e\u003c/a\u003e\n##### [sys-wait](sys-wait)\n\nBash script to check and wait for various system conditions,\nfiles, processes or thresholds like load average or PSI values.\n\nRandom examples:\n\n    % sys-wait -l 3 \u0026\u0026 run-less-heavy-task\n    % sys-wait --load15 5 \u0026\u0026 run-next-heavy-task\n    % sys-wait -f /some/file/appeared \u0026\u0026 process-file\n    % sys-wait -F /file/to-be-removed \u0026\u0026 run-stuff\n\nHelps to avoid writing those annoyingly-common\n`while :; do some-check || break; sleep 60; done; run-other-stuff`\nwhen something heavy/long is already running and you just don't\nhave the heart to break and reschedule it properly.\n\nMostly used to need for pgrep in a loop, but these days util-linux includes\npidwait binary, which does the job without this wrapper.\n\n\u003ca name=hdr-yt-feed-to-email\u003e\u003c/a\u003e\n##### [yt-feed-to-email](yt-feed-to-email)\n\nPython + [feedparser] RSS-to-email notification script for YouTube RSS feeds.\n\nCan process OPML of current YT subscriptions (from\n\u003chttps://www.youtube.com/subscription_manager?action_takeout=1\u003e )\nor work with one-per-line list of channel/video RSS feed links.\n\nRemembers last feed state(s) via auto-rotating log, uses [EWMA]\nto calculate delay between checks based on feed update interval.\n\nUseful to keep track of YT channel updates via read/unread status in some\ndedicated mailbox folder, and click-open video links from there in mpv,\nlike one could before Aug 2020 when google decided to stop sending all update\nnotification emails on that platform.\n\n[feedparser]: https://pythonhosted.org/feedparser/\n[EWMA]: https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average\n\n\u003ca name=hdr-color-b64sort\u003e\u003c/a\u003e\n##### [color-b64sort](color-b64sort)\n\nTool to filter, sort and compress list of colors - aka color palette - into\nbase64, to then use as a compact blob in visualization scripts easily.\n\n- Input: a list of hex-encoded colors, separated by any spaces/newlines.\n\n- Filtering:\n\n  Removes colors too close to specified background color\n  (using specified Delta E CIE 2000 color-diff threshold).\n\n  Compares colors all-to-all, and removes ones that are too close to each other,\n  with a similar configurable threshold.\n\n- Ordering:\n\n  Picks next color based on min(deltas-with-others) value, to get the most\n  distinct color on every step.\n\n  This is further configured by using higher weights of min(deltas-with-n-last)\n  colors, so that next pick ends up being as distinct as possible from N ones\n  that are right before it first, and then the rest of them.\n\n  Current default for `-k/--sort-delta-keys` \"weight:count\" list is \"0.3:5 0.2:10\n  0.1:20\", with leftover 0.4 weight used for min(deltas-with-all-picked) value.\n\n- Output:\n\n  Urlsafe-base64 of concatenated 3-byte color values in RGB order,\n  instead of more bulky \"lines of hex-encoded colors\" or other color-spec types,\n  to hardcode without taking too much space.\n\nIntended use it to have output color list of 50+ values, and then pick them in\norder (for chart lines, tree branches, table row/cell backgrounds, etc), which\nshould return most distinctive colors first, without resorting to repetition as\nquickly as with e.g. D3.js fixed 10/20-color palettes.\n\nThere are many great tools like [\"i want hue\"] that can be used to generate input\ncolor list for this script, with features like accounting for color blindness types,\nbut it can be just a sequence of points from any nice gradient too - input\nordering or similarity should not matter.\n\nIt's a small python script, which uses [colormath] module for Delta E CIE 2000\ncolor-diff calculations.\nCan take some time to run with long lists due to how all\\*all combinatorics work,\nbut using pypy instead of cpython can speed that up a lot.\n\n[\"i want hue\"]: https://medialab.github.io/iwanthue/\n[colormath]: https://python-colormath.readthedocs.io/\n\n\u003ca name=hdr-svg-tweak\u003e\u003c/a\u003e\n##### [svg-tweak](svg-tweak)\n\nSmall python script to change SVG files, according to specified options.\n\nFor example, if an image viewer displays transparent SVG with back text on a black\nbackground (as one solid-black rectangle), `svg-tweak -b '#fff' file.svg` can fix it.\n\nSVGs are XML text, so aren't difficult to change like that, but old unix cli tools\nlike sed and awk aren't great for that, and tend to require a bunch of extra logic.\n\n\u003ca name=hdr-unix-socket-links\u003e\u003c/a\u003e\n##### [unix-socket-links](unix-socket-links)\n\nPython wrapper around `ss -xp` output, processing disjointed unix socket\nconnection table (with pids on only one end of those), into more readable\naggregated `\u003csocket-path\u003e :: \u003clistening-pid\u003e :: \u003cclients...\u003e` list.\n\n`ss -xp src \u003csocket\u003e` is closest to this functionality, but doesn't actually list\nclients connected there, e.g. for X11 socket it lists same Xorg process uselessly\nfor each connection, instead of actual X apps connected to that socket.\n\nUse-case is to quickly check what's connected to some socket path\n(which maybe you don't remember exactly), by printing a short list of all\nof them with listener/client pids, when some connection hangs or ssh-agent\nasks for fido2 touch-check unexpectedly.\n\nHas more human-readable `-p/--pretty` mode and more traditional disaggregated\n`-c/--conns` mode for listing specific connections instead of just processes.\n\nSee [\"List connected processes for unix sockets\" blog post] for some usage examples.\n\n[\"List connected processes for unix sockets\" blog post]:\n  https://blog.fraggod.net/2024/08/06/list-connected-processes-for-unix-sockets-on-linux.html\n\n\u003ca name=hdr-tcpdump-translate\u003e\u003c/a\u003e\n##### [tcpdump-translate](tcpdump-translate)\n\nWrapper script for running `tcpdump -ln` (unbuffered lines, no dns), to translate,\ncolor-highlight and optionally filter-by specified addresses and network prefixes.\n\nThere are couple images showing what it does in [\"Adding color to tcpdump\" blog post].\n\n[\"Adding color to tcpdump\" blog post]:\n  https://blog.fraggod.net/2024/09/30/adding-color-to-tcpdump-makes-a-ton-of-difference.html\n\nIntended use is to match known hosts or networks in the output, while leaving\nall other addresses intact, without going to DNS PTR records or anything like that.\n\nFor example, with the following `ipv6-debug.tt` file:\n```\n# \"\u003cprefix/net/addr\u003e \u003creplacement\u003e [!\u003chighlight\u003e]\" specs, newline/comma separated\n# Exact-match full address should end with \"/\". Example: 1.2 mynet, 1.2.3.4/ myaddr\n\n2a01:4f8:c27:34c2:   A.net:\n2a01:4f8:c27:34c2::2/ [A]\n\n2a01:4f8:c27:34c2:8341:8768:e26:83ff/ [A.ns] !red\n\n2a02:13d1:22:6a0      B.net\n2a02:13d1:22:6a01::1/ [B]\n\n2a02:13d1:22:6a00:2a10:6f67:8c0:60ae/ [B.host-X] !bold-green\n2a02:13d1:22:6a00:de8a:12c8:e85:235f/ [B.laptop] !bold-bright-yellow\n\n127.0.0. lo4., :: lo6.\n```\n\nAnd then running e.g. `tcpdump -i eth0 | ./tcpdump-translate -m ipv6-debug.tt`\nwill produce translated output (also truncated to terminal width by default):\n```\n11:40:00.641680 IP6 A.net:8341:865e:e26:8401.31788 \u003e [B.laptop].31788: UDP, length 32\n11:41:49.868243 IP6 [A.ns].31788 \u003e B.net0:de8c::28f1.31788: UDP, length 148\n11:41:51.148385 IP6 [A.ns].31788 \u003e B.net0:de8c::28f2.31788: UDP, length 148\n...\n11:42:23.735140 IP6 [A.ns].31788 \u003e [B.laptop].31788: UDP, length 148\n11:42:24.801590 IP6 [A.ns].31788 \u003e [B].11446: UDP, length 148\n11:42:26.286887 IP6 [B.host-X].31788 \u003e [A.ns].31788: UDP, length 32\n11:42:26.287739 IP6 [B.host-X].31788 \u003e [A.ns].31788: UDP, length 148\n11:42:26.288301 IP6 [A.ns].31788 \u003e [B.host-X].31788: UDP, length 92\n11:42:26.350673 IP6 [B.host-X].31788 \u003e [A.ns].31788: UDP, length 32\n11:42:29.068373 IP6 [A.ns].31788 \u003e [B.laptop].31788: UDP, length 148\n11:42:29.573134 IP6 [A.ns].47504 \u003e [B].80: Flags [S], seq 3249847667, win 33120,\n11:42:29.638883 IP6 [B].80 \u003e [A.ns].47504: Flags [S.], seq 271826300, ack 324984\n11:42:29.639081 IP6 [A.ns].47504 \u003e [B].80: Flags [.], ack 1, win 259, options\n...\n11:42:29.705541 IP6 [A.ns].47504 \u003e [B].80: Flags [F.], seq 75, ack 375, win 257,\n11:42:29.770506 IP6 [B].80 \u003e [A.ns].47504: Flags [F.], seq 375, ack 76, win 251,\n11:42:29.770583 IP6 [A.ns].47504 \u003e [B].80: Flags [.], ack 376, win 257, options\n11:42:29.921720 IP6 [A.ns].31788 \u003e [B].11446: UDP, length 148\n```\n\nWhere replacements are done either for full addresses or their string prefixes\n(not CIDR prefixes, simple string match-replace).\n\nWithout this, IPv6es in output above","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmk-fg%2Ffgtk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmk-fg%2Ffgtk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmk-fg%2Ffgtk/lists"}