{"id":19388310,"url":"https://github.com/mk-fg/kelp","last_synced_at":"2025-02-24T19:57:53.887Z","repository":{"id":144996195,"uuid":"196112456","full_name":"mk-fg/kelp","owner":"mk-fg","description":"Personal ircd for hosting various asyncio notification bots","archived":false,"fork":false,"pushed_at":"2025-01-21T02:26:57.000Z","size":265,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-21T03:23:51.056Z","etag":null,"topics":["asyncio","automation","events","irc-bot","ircd","monitoring","notification","personal-assistant","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"wtfpl","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mk-fg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-10T02:01:52.000Z","updated_at":"2025-01-21T02:27:00.000Z","dependencies_parsed_at":"2024-05-13T01:35:27.635Z","dependency_job_id":"4500d889-be75-4e42-bbe8-5751858f1aac","html_url":"https://github.com/mk-fg/kelp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fkelp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fkelp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fkelp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mk-fg%2Fkelp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mk-fg","download_url":"https://codeload.github.com/mk-fg/kelp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240549434,"owners_count":19819131,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","automation","events","irc-bot","ircd","monitoring","notification","personal-assistant","python"],"created_at":"2024-11-10T10:12:26.142Z","updated_at":"2025-02-24T19:57:53.879Z","avatar_url":"https://github.com/mk-fg.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kelp IRC Daemon\n\nPersonal ircd for hosting various asyncio notification bots.\n\nTable of Contents\n\n- [Description](#hdr-description)\n- [Usage](#hdr-usage)\n- [\"blade\" plugins](#hdr-_blade__plugins)\n\n    - [udp-report-sink]\n    - [logtail](#hdr-logtail)\n\n- [Requirements](#hdr-requirements)\n- [Misc Features](#hdr-misc_features)\n\n    - [Multiple Config Files](#hdr-multiple_config_files)\n    - [Channel Commands](#hdr-channel_commands)\n    - [Monitor Channel](#hdr-monitor_channel)\n    - [Aliases](#hdr-aliases)\n\n![kelp forest stacks][]\n\n[udp-report-sink]: #hdr-udp-report-sink\n[kelp forest stacks]: kelp.jpg\n\n\n\u003ca name=hdr-description\u003e\u003c/a\u003e\n## Description\n\nPython3/asyncio daemon to run personal ircd for hosting various other asyncio\ncode - bots that'd populate its irc channels, most likely with line-based\nnotifications and reports, as that's what this format is good for.\n\nIt's very much personal helper scripts, so probably only useful to me as-is,\nbut feel free to use anything here that might be of value otherwise.\n\nOriginally based on [rdircd] irc-daemon part,\nand intended to replace [earlier twisted-based bot].\n\nRepository URLs:\n\n- \u003chttps://github.com/mk-fg/kelp\u003e\n- \u003chttps://codeberg.org/mk-fg/kelp\u003e\n- \u003chttps://fraggod.net/code/git/kelp\u003e\n\n[rdircd]: https://github.com/mk-fg/reliable-discord-client-irc-daemon\n[earlier twisted-based bot]: https://github.com/mk-fg/bordercamp-irc-bot\n\n\n\u003ca name=hdr-usage\u003e\u003c/a\u003e\n## Usage\n\nThere are no main script dependencies to install, beyond basic python 3.x.\n\nCreate configuration file with ircd auth credentials and any other settings in\n`~/.kelp.ini` (see all the --conf\\* opts wrt these):\n\n``` ini\n[irc]\npassword = hunter2\n```\n\nNote: IRC password can be omitted, but be sure to firewall that port from\neverything in the system then, which is also a good idea to do anyway.\n\nStart kelp daemon: `./kelp --debug`\n\nConnect IRC client to `localhost:6667` (see `./kelp --conf-dump-defaults`\nor `-i/--irc-bind` option for using diff host/port).\n\nRun `/list` to see channels for all running bots, services and such,\nas well as server control channels (#kelp.control, #kelp.debug)\nand catch-all #kelp.monitor, see topics for more info on these.\n\n\n\u003ca name=hdr-_blade__plugins\u003e\u003c/a\u003e\n## \"blade\" plugins\n\nThey are actual point of the whole thing.\n\nEnabled like this:\n\n``` ini\n[blades]\nenabled = udp-report-sink some-other-script\n```\n\nSpecified .py files loaded from `dir = ...` (default \"blades\") and last definition\nthere is initialized as an async context manager, being passed KelpBladeInterface object.\n\nEach gets short base64 prefix for all channels it generates (based on name),\nand can use these (send-to, read, query names) separately from any other loaded blade-scripts.\nCan be configured via separate section(s) in the ini file(s).\n\nSee scripts in \"blades\" dir and KelpBladeInterface object for any implementation details.\n\n\n\u003ca name=hdr-udp-report-sink\u003e\u003c/a\u003e\n### udp-report-sink\n\nPlugin for remote logging/reporting.\n\nIntended to receive occasional maybe-multiline log error/warning reports from\nremote sources to feed them into irc channel(s) as notifications, picking\ndestination channel based on nacl crypto_box pubkey used for decryption.\n\nIn addition can create/track heartbeat timers, and warn about local/remote error\ncounter mismatches (e.g. due to failed delivery), for persistent remote endpoints.\nDoes auth-encryption via [libnacl]/[pynacl] [libsodium] wrappers.\n\nConfig sections:\n\n- `udp-report-sink` - see UDPRSConf in [udp-report-sink.py].\n\n- `udp-report-sink-chans` - channel to source nacl crypto_box pubkeys mapping.\n\n    Each key is a channel name, values are space-separated crypto_box pubkeys or\n    @name aliases (from `udp-report-sink-keys` secion) for all potential sources\n    which will be dumped into this channel.\n\n    Special \"{chan}.topic\" and \"{chan}.nick\" keys can be used to specify\n    topic/nick for each channel, otherwise defauls from UDPRSConf will be used.\n\n- `udp-report-sink-keys`\n\n    `\u003cname\u003e = \u003cb64(pk)\u003e` aliases for source pubkeys, so that any reports about\n    these (e.g. missing heartbeats, error count, etc) will have easy-to-read\n    name instead of hard-to-remember keys.\n\nExample config for couple receiver pubkeys into two separate IRC channels:\n\n``` ini\n[udp-report-sink]\nhost = 0.0.0.0:1234\n;; uid-mask bits below should results in a\n;;  pkt[:8] \u0026 0x1008104104104104 == 0x100004100100 filter\n;; Such filtering is to avoid auth-checking or logging random udp noise\nuid-mask-intervals = 3, 9, 7, 6\nuid-mask-bits = --x--xx-x-\ncb_key = _p0ZbIHfK86H263_DBvaAbyrglrmqhcY0dOBppyPmgU=\n\n[udp-report-sink-chans]\n\ntest = @test-key @logger-A\ntest.topic = Reports from A and misc testing\ntest.nick = crashey\n\nmain = @logger-A @logger-B @logger-C\nmain.topic = Reports from A, B and C\nmain.nick = reporterbot\n\n[udp-report-sink-keys]\ntest-key = Msf_VdIGWquWN2SwCs9A4hDaE9rBUSkoxWiiOiLCQkY=\nlogger-A = YJkoWjFc7yOyXP23PLVLDy0Izl0B4Hu/Z8eb6PyahUk=\nlogger-B = 0Fhfk1ujYvrEUkJ8VaO9YySc02Q/wE2c9vi597bIBHI=\nlogger-C = +JqMz5Pzj2yjAD/XYDoaOHmnvgPe2PwjkLeJoDTnqWM=\n```\n\nAll crypto keys can be generated via `wg genkey` (wireguard also uses\ncurve-25519 keys) or base64-encoding 32 bytes from `/dev/urandom`.\n\nSee [udp-report-send-test.py] for an example of a simple sender script.\n\n[pynacl]: https://pynacl.readthedocs.io/\n[libnacl]: https://libnacl.readthedocs.io/\n[libsodium]: https://doc.libsodium.org/\n[udp-report-sink.py]: blades/udp-report-sink.py\n[udp-report-send-test.py]: blades/udp-report-send-test.py\n\n\n\u003ca name=hdr-logtail\u003e\u003c/a\u003e\n## logtail\n\nPlugin for tailing a log file (lines of text) in an efficient and reliable\nmanner into irc channel, remembering last-reported position and handling\nrename-rotation (but NOT truncation).\n\nUses inotify to monitor file(s) for updates and rotation, storing position\nand a checksum of last N bytes to a state-file with some rate-limiting\nfor reads to batch-process frequent messages.\n\nTailed files are assumed to become static after rotation (filename change)\nwithin specified timeout, after which they're closed and no longer monitored.\n\nConfig sections:\n\n- `logtail` - see LogtailConf in [logtail.py].\n\n- `logtail-files` - mapping between monitored files and channels.\n\n    Each key is a channel name, values are space-separated file paths to monitor.\n    Weird filenames can be urlencoded (decoded via urllib.parse.unquote).\n\n    Special \"{chan}.topic\" and \"{chan}.nick\" keys can be used to specify\n    topic/nick for each channel, otherwise defauls from LogtailConf will be used.\n\n- `logtail-files-proc` - regexp-rules for processing individual log lines.\n\n    All rule keys start with arbitrary prefix to group multiple keys, and\n    dot-separated suffix after that determines purpose of the value,\n    similar to \".topic\" and \".nick\" for channels above.\n\n    Every rule must have \".file\" value to set which file to apply it to,\n    and \".re\" with python regexp to match each processed line.\n    Rules are applied in order they appear in and can affect each other.\n\n    Full list of supported rule-suffixes:\n\n    - `file` (required) - path used in `logtail-files` section to apply this rule to.\n\n    - `re` (required) - regexp to match against each line after str.rstrip()\n        (no tailing whitespace, newlines and such) to check if it should be affected\n        by this rule. In python's \"re\" module format.\n\n    - `sub` - substitution pattern, second argument to python's re.sub().\n\n    - `rate-tb` - token-bucket rate-limit applied to affected messages.\n\n        Value format: `{ interval_seconds: float | float_a/float_b }[:burst_float]`\\\n        Examples: 1/4:5 (interval=0.25s, rate=4/s, burst=5), 5, 0.5:10, 20:30, 1/2.\n\n        Lines that go over the limit are skipped, with system message printed between\n        last passed and first skipped message to indicate when rate-limiting starts.\n\n    - `filter` - either \"whitelist\" (default) or \"blacklist\" to silently drop\n        either all non-matching or matching lines respectively.\n\n    See example below for more info.\n\nExample config for a couple logs to a #system channel with some parameters:\n\n``` ini\n[logtail]\nstate-dir = /var/lib/kelp\nread-interval-min = 0.3\npost-rotate-timeout = 1.0\n\n[logtail-files]\nsystem = /var/log/kmsg.log /var/log/syslog.log\nsystem.topic = System log tailer channel\nsystem.nick = mon\n\n[logtail-files-proc]\n\nsyslog-clean.file = /var/log/syslog.log\nsyslog-clean.re = ^[-\\d]{10}T[:\\d]{6}(\\.\\d+)?([-+]\\d{2}:\\d{2})? (?P\u003cchan\u003e[\\w.]+)(\u003c\\d+\u003e)? (?P\u003cmsg\u003e.*)$\nsyslog-clean.sub = \\g\u003cchan\u003e \\g\u003cmsg\u003e\n\nsyslog-selfnoise.file = /var/log/syslog.log\nsyslog-selfnoise.re = \\skelp\\[(\\d+|-)\\]@\\w+:\\s\nsyslog-selfnoise.rate-tb = 20\n```\n\nFiles can be used as simple persistent queues for text messages from anywhere,\nand this tailer allows to use those for irc notifications.\n\n[logtail.py]: blades/logtail.py\n\n\n\u003ca name=hdr-requirements\u003e\u003c/a\u003e\n## Requirements\n\n* [Python 3.8+](https://python.org/)\n* [udp-report-sink] - [libnacl] or [pynacl] (whichever can be imported)\n\n\n\u003ca name=hdr-misc_features\u003e\u003c/a\u003e\n## Misc Features\n\nNotes on various optional and less obvious features are collected here.\\\nSee \"Usage\" section for a more general information.\n\n\u003ca name=hdr-multiple_config_files\u003e\u003c/a\u003e\n### Multiple Config Files\n\nMultiple ini files can be specified with `-c/--conf` option,\noverriding each other in that order.\n\nLast one will be updated wrt \\[state\\] and similar runtime stuff,\nso it can be useful to specify persistent config with auth and options,\nand separate (initially empty) one for such dynamic state.\n\nE.g. `./kelp -c config.ini -c state.ini` will do that.\\\nAdding `--conf-dump` option will print resulting ini assembled from all these.\n\nFrequent state timestamp updates are done in-place (small fixed-length values),\nbut checking ctime before writes, so should be safe to tweak any of these files\nanytime anyway.\n\nIf plugin stores runtime data in ini files, that should be mentioned in its docs.\n\n\u003ca name=hdr-channel_commands\u003e\u003c/a\u003e\n### Channel Commands\n\nIn special channels like #kelp.control and #kelp.debug:\nsend \"h\" or \"help\", see topic there.\n\nPlugins can react to user messages as well, in their own ways,\nwhich should be documented, if any.\n\n\u003ca name=hdr-monitor_channel\u003e\u003c/a\u003e\n### Monitor channel\n\n#kelp.monitor gets all messages from all channels, and can be used as\none read-only catch-all place to monitor notifications from all plugins.\n\n\u003ca name=hdr-aliases\u003e\u003c/a\u003e\n### Aliases\n\nCan be defined in the config file to replace hash-based IDs with something\nmore easily readable:\n\n``` ini\n[aliases]\nblade.csug = logs\n```\n\n(to turn e.g. #csug.system into #logs.system, and same for other channels of that plugin)\n\nCurrently only implemented for Blade UIDs in IRC channel names.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmk-fg%2Fkelp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmk-fg%2Fkelp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmk-fg%2Fkelp/lists"}